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Abstract 


Intelligence  analysis  provides  important  informational  support  to  civilian  and  military  decision 
makers.  Recent  intelligence  failures  of  Canada’s  allies  have  been  attributed  mostly  to  cognitive, 
social,  and  organizational  deficits  and  biases  of  individual  analysts  and  intelligence  agencies. 
Such  attributions  call  for  a  comprehensive  examination  of  intelligence  production  from  the  socio- 
psychological  perspective.  The  present  report  discusses  findings  from  interviews  conducted  with 
Canadian  managers  of  intelligence  analysts.  The  interviewed  managers  identified  a  number  of 
pertinent  issues  in  the  intelligence  production  process  that  may  be  explicated  through  the 
application  of  the  behavioural  sciences’  accumulated  knowledge  and  methodology.  The 
identified  issues  are  discussed  in  light  of  the  intelligence  studies  and  behavioural  sciences 
literature,  and  a  roadmap  for  the  behavioural  sciences  research  program  in  support  of  the 
intelligence  function  is  outlined. 


Resume 


L’ analyse  du  renseignement  offre  un  important  soutien  informationnel  aux  decideurs  civils  et 
militaires.  Les  recents  echecs  d’ allies  du  Canada  dans  le  domaine  du  renseignement  ont  ete 
principalement  attribues  a  des  lacunes  cognitives,  sociales  et  organisationnelles,  ainsi  qu’aux 
prejuges  des  analystes  et  des  organismes  du  renseignement.  Un  tel  constat  exige  la  tenue  d’un 
examen  en  profondeur  de  la  production  du  renseignement  d’un  point  de  vue  socio-psychologique. 
Le  present  rapport  porte  sur  les  conclusions  tirees  des  entrevues  menees  aupres  de  gestionnaires 
canadiens  d’ analystes  du  renseignement.  Les  gestionnaires  interviewes  ont  degage  un  certain 
nombre  de  problemes  pertinents  dans  le  processus  de  production  du  renseignement  que  l’on 
pourrait  expliquer  par  la  mise  en  application  des  connaissances  et  des  methodes  acquises  dans  le 
domaine  des  sciences  du  comportement.  Les  problemes  releves  sont  abordes  sur  la  base  d’etudes 
sur  le  renseignement  et  de  publications  sur  les  sciences  du  comportement.  Le  rapport  contient 
egalement  Taperqu  de  la  feuille  de  route  d’un  programme  de  recherche  en  sciences  du 
comportement  qui  appuierait  la  fonction  du  renseignement. 
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Executive  summary 


Issues  in  Intelligence  Production:  Summary  of  interviews  with 
Canadian  managers  of  intelligence  analysts 

Derbentseva,  N.;  McLellan,  L.;  Mandel,  D.R.;  DRDC  Toronto  TR  2010-144; 
Defence  R&D  Canada  -  Toronto;  December  2010. 

Introduction  or  background:  Intelligence  analysis  is  an  important  state  function  that  informs 
and  supports  policy  and  command-and-control  decision  making.  Intelligence  analysts  seek  to 
reduce  uncertainty  and  improve  decision  quality  for  intelligence  consumers  by  employing  their 
analytic  skills  to  derive  judgments  from  available  information,  much  of  which  is  uncertain  and 
which  may  also  conceal  deception.  Because  intelligence  analysis  predominantly  relies  on  human 
reasoning  and  judgment,  there  is  considerable  opportunity  for  the  behavioural  sciences  to  be 
applied  to  the  task  of  better  understanding  and  ultimately  improving  intelligence  analysis. 
However,  the  open-source,  unclassified  literature  on  the  application  of  behavioural  science  to 
intelligence  analysis  is  scarce,  reflecting  the  fact  that  there  is  a  paucity  of  applied  behavioural 
science  in  support  of  intelligence.  In  the  present  technical  report,  an  investigative  interview  study 
is  described.  The  study  aimed  to  identify  pertinent  issues  in  intelligence  analysis  and  to 
develop  a  roadmap  for  future  behavioural  science  research  that  could  support  the  intelligence 
analysis  function. 

Method:  The  present  study  was  conducted  by  members  of  the  Thinking,  Risk,  and  Intelligence 
Group  (TRIG)  in  the  Adversarial  Intent  Section  at  Defence  R&D  Canada  -  Toronto,  as  part  of  the 
preliminary  research  for  an  Applied  Research  Program  project  on  understanding  and  augmenting 
human  analytic  capabilities  for  intelligence,  under  the  sponsorship  of  the  Chief  of  Defence 
Intelligence  (CDI).  TRIG  researchers  conducted  semi-structured  interviews  with  seven 
intelligence  managers  from  two  Canadian  intelligence  organizations  -  CDI  and  International 
Assessment  Staff  of  the  Privy  Council  Office.  Interview  discussions  covered  a  variety  of  topics: 
Analysts’  tasks  and  analytic  process,  challenges  in  intelligence  analysis,  essential  skills  required 
for  intelligence  analysis,  the  selection  process  of  analysts,  training,  analytic  tools,  managers’  roles 
and  challenges,  and  characteristics  of  the  Canadian  intelligence  community  at  large. 

Results:  Managers’  responses  identified  several  areas  for  further  research,  including 
the  following: 

Study  of  cognitive  processes  involved  in  information  search,  evaluation,  and  analysis: 
Behavioural  science  research  could  be  instrumental  in  revealing  how  different  formulations  of  an 
intelligence  question  may  affect  judgment;  understanding  processes  that  underlie  the  tendency 
among  analysts  and  consumers  to  overly  rely  on  classified  information  based  solely  on  the  fact 
that  it  is  classified;  investigating  the  role  and  the  impact  of  a  number  of  cognitive  biases  (such  as 
confirmation  bias,  mirror  imaging,  and  status  quo),  which  may  impair  analysts’  judgment; 
understanding  the  mechanisms  for  coping  with  information  overload  and  tradeoffs  between  the 
continuing  search  for  new  information  and  the  analysis  of  the  available  evidence;  and  identifying 
roots  and  devising  means  for  dealing  with  such  analysts’  behaviours  as  decision  avoidance, 
defensiveness  and  unwillingness  to  accept  other  perspectives  on  the  issue  they  analyzed. 
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Identifying  essential  skills  required  for  intelligence  analysis  and  developing  skill  assessment 
tools:  Research  on  individual  differences  accumulated  in  the  behavioural  sciences  could  be 
instrumental  in  identifying  the  set  of  essential,  inherent  and  acquired  abilities  that  are  required  for 
intelligence  analysis.  Identifying  these  skills  and  abilities  and  developing  reliable  measures  of 
these  attributes  may  improve  the  analyst  selection  and  performance  evaluation  processes,  and 
support  the  development  of  intelligence  analysis  training  programs. 

Systematic  evaluation  of  analytic  tools  and  techniques:  Systematic  scientific  evaluation  of  the 
available  tools  and  techniques  can  provide  a  better  understanding  of  their  drawbacks  and  benefits 
to  support  analysis  and  the  conditions  required  for  their  application. 

Developing  methods  for  evaluating  training  effectiveness:  Objective  evaluation  of  intelligence 
analyst  training  can  support  the  development  of  training  programs  and  facilitate  transfer  of 
training  to  the  workplace. 

Surveying  current  knowledge  management  practices  and  needs:  Further  survey  research  along 
such  lines  could  be  informative  to  the  community  in  dealing  with  knowledge  preservation  and 
transfer  issues  arising  from  such  organizational  challenges  as  turnover  and  inadequate  staffing. 

Investigating  new  methods  for  product  and  performance  evaluation:  Due  to  the  lack  of 
unambiguous  evaluation  criteria  and  variability  of  analytic  standards  and  requirements, 
intelligence  product  evaluation  mostly  relies  on  managers’  subjective  judgment.  The  community 
can  benefit  from  the  development  of  more  objective  means  for  evaluating  analysts’  performance, 
such  as  accuracy  of  their  aggregate  judgments. 

Organizational  analysis:  Organizational  analysis  can  be  instrumental  in  identifying  the 
sources  of  issues  such  as  inadequate  staffing,  time  pressure,  turnover  (mainly  of  military 
analytic  personnel),  lack  of  feedback  on  final  intelligence  products,  and  barriers  in  inter¬ 
agency  and  inter-departmental  information  sharing.  All  of  these  issues  can  impact  analysts’ 
productivity  and  contribute  to  the  disruption  of  organizational  processes,  loss  of  expertise,  and 
organizational  memory. 

Significance:  The  present  report  summarises  results  of  a  unique  interview  study  with  Canadian 
managers  of  intelligence  analysts.  Various  issues  identified  by  the  managers  are  discussed  in  light 
of  both  intelligence  studies  and  the  behavioural  sciences  literature.  Based  on  input  from  these 
intelligence  managers,  the  report  outlines  a  roadmap  for  behavioural  sciences  research  in  support 
of  the  intelligence  analysis  function.  Scientific  research  in  the  identified  areas  would  improve  our 
understanding  of  the  outlined  issues  and  could  provide  valuable  insights  into,  and  means  of, 
augmenting  human  performance  in  intelligence  production. 

Future  plans:  The  identified  areas  for  research  and  development  encompass  many  issues  and 
span  several  disciplines.  Research  efforts  in  several  of  the  identified  areas  are  already  underway 
in  TRIG,  such  as  calibration  of  analysts’  judgment  accuracy,  communication  of  verbal  probability 
estimations,  the  role  of  reliability  and  diagnosticity  of  information  in  decision  making,  belief 
revision  based  on  new  evidence,  evaluating  the  effectiveness  of  cultural  sensitivity  training,  the 
relationship  between  individual  differences  and  accuracy  and  coherence  in  decision  making,  the 
impact  of  question  framing  on  probability  estimation,  and  means  for  visualizing  knowledge  and 
information.  However,  it  would  be  impossible  to  address  all  of  the  questions  discussed  in  the 
report  within  a  scope  of  a  single  project.  The  research  team  will  identify  priorities  for  future  work 
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through  continuous  and  close  interaction  with  members  of  the  Canadian  intelligence  community 
to  ensure  that  the  research  efforts  address  the  needs  of  that  community,  and  especially  the  needs 
of  our  sponsors  and  partners. 
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Introduction  ou  contexte:  L’analyse  du  renseignement  est  une  importante  fonction  d’etat  qui 
sert  de  base  et  qui  contribue  a  la  prise  de  decisions  en  matiere  de  politique  et  de  commandement 
et  controle.  Les  analystes  du  renseignement  visent  a  reduire  les  incertitudes  et  a  ameliorer  la 
qualite  des  decisions  prises  par  les  utilisateurs  du  renseignement  en  mettant  a  profit  leurs 
capacites  d’ analyse  pour  formuler  des  jugements  a  partir  des  informations  disponibles,  qui  sont 
pour  la  plupart  incertaines  et  susceptibles  d’etre  fausses.  Comme  1’ analyse  du  renseignement 
repose  principalement  sur  le  raisonnement  et  le  jugement  humains,  il  s’agit  d’une  excellente 
occasion  d’appliquer  les  principes  des  sciences  du  comportement  dans  le  but  de  mieux 
comprendre  1’ analyse  du  renseignement  et,  en  fin  de  compte,  de  1’ ameliorer.  Cependant,  la 
documentation  non  classifiee  et  en  libre  acces  sur  1’ application  des  sciences  du  comportement  a 
l’analyse  du  renseignement  est  peu  abondante,  ce  qui  temoigne  du  manque  de  recherche  en 
sciences  appliquees  du  comportement  a  l’appui  du  renseignement.  Le  present  rapport  technique 
contient  la  description  d’une  etude  approfondie  realisee  par  le  biais  d’entrevues.  Le  but  de  T  etude 
etait  de  degager  des  problemes  pertinents  dans  le  domaine  de  1’ analyse  du  renseignement  et 
d’elaborer  la  feuille  de  route  d’un  futur  programme  de  recherche  en  sciences  du  comportement 
qui  appuierait  la  fonction  du  renseignement. 

Methode  :  La  presente  etude  a  ete  menee  par  des  membres  du  Groupe  reflexion,  risque  et 
renseignement  (TRIG)  de  la  Section  des  intentions  antagonistes  de  R  &  D  pour  la  defense  Canada 
-  Toronto,  dans  le  cadre  de  la  recherche  preliminaire  sur  le  projet  Programme  de  recherches 
appliquees,  qui  porte  sur  la  comprehension  et  1’ augmentation  des  capacites  humaines  d’ analyse 
du  renseignement  et  qui  est  parraine  par  le  Chef  du  renseignement  de  la  defense.  Les  chercheurs 
du  TRIG  ont  mene  des  entrevues  semi-structurees  aupres  de  sept  gestionnaires  d’ analystes  du 
renseignement  provenant  de  deux  organismes  du  renseignement  canadiens  -  le  Chef  du 
renseignement  de  la  defense  et  le  Bureau  de  revaluation  internationale  du  Bureau  du  Conseil 
prive.  Les  entrevues  ont  porte  sur  divers  sujets  :  les  taches  et  le  processus  analytique  des 
analystes,  les  difficultes  de  1’ analyse  du  renseignement,  les  competences  essentielles  requises 
pour  realiser  des  analyses  du  renseignement,  le  processus  de  selection  des  analystes,  la  formation, 
les  outils  d’ analyse,  le  role  et  les  defis  des  gestionnaires  et  les  caracteristiques  de  la  communaute 
canadienne  du  renseignement  dans  son  ensemble. 

Resultats:  Les  reponses  des  gestionnaires  ont  permis  de  cerner  plusieurs  aspects  necessitant  une 
recherche  plus  approfondie,  a  savoir  les  suivants  : 

Etude  des  processus  cognitifs  intervenant  dans  la  recherche,  revaluation  et  Vanalyse 
d’ informations  :  La  recherche  en  sciences  du  comportement  pourrait  contribuer  a  reveler 
comment  differentes  formulations  d’une  question  sur  le  renseignement  peuvent  avoir  une 
incidence  sur  le  jugement;  a  comprendre  le  processus  qui  est  a  l’origine  de  la  tendance,  chez  les 
analystes  et  les  utilisateurs  du  renseignement,  a  trop  se  fier  a  Tinformation  classifiee  du  seul  fait 
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qu’elle  soit  classifiee;  a  etudier  le  role  et  les  repercussions  de  differents  prejuges  cognitifs  (p.  ex., 
prejuge  fonde  sur  la  confirmation,  image-miroir  et  statu  quo)  pouvant  alterer  le  jugement  des 
analystes;  a  comprendre  les  mecanismes  permettant  de  faire  face  a  la  surdose  d’information  et 
aux  compromis  a  faire  entre  la  recherche  de  nouvelles  informations  et  1’ analyse  des  donnees 
disponibles;  et  a  determiner  les  origines  de  certains  comportements  observes  chez  les  analystes, 
comme  le  fait  d’eviter  de  prendre  une  decision,  1’ attitude  defensive  et  la  reticence  a  accepter 
d’autres  points  de  vue  sur  ce  qu’ils  sont  en  train  d’analyser,  ainsi  qu’a  elaborer  des  methodes  pour 
faire  face  a  ces  comportements. 

Identification  des  competences  essentielles  requises  pour  realiser  des  ancdyses  du  renseignement 
et  elaboration  d’outils  pour  revaluation  des  competences :  Les  recherches  sur  les  differences 
individuelles  accumulees  en  sciences  du  comportement  pourraient  jouer  un  role  important  dans 
l’etablissement  de  1’ ensemble  des  competences  essentielles  inherentes  et  acquises  necessaires 
pour  realiser  des  analyses  du  renseignement.  L’identification  de  ces  competences  et  habiletes  et  la 
definition  de  mesures  fiables  de  ces  attributs  pourraient  permettre  d’ameliorer  les  processus  de 
selection  et  d’ evaluation  du  rendement  des  analystes,  ainsi  que  d’appuyer  la  mise  sur  pied  de 
programmes  de  formation  en  analyse  du  renseignement. 

Evaluation  systematique  des  outils  et  des  techniques  d’analyse  :  L’evaluation  scientifique 
systematique  des  divers  outils  et  techniques  d’analyse  disponibles  peut  permettre  de  mieux  en 
comprendre  les  avantages  et  les  inconvenients  et  de  connaitre  les  conditions  dans  lesquelles  ils 
doivent  etre  utilises. 

Elaboration  de  methodes  pour  evaluer  I’efficacite  de  la  formation  :  Une  evaluation  objective  de 
la  formation  des  analystes  du  renseignement  peut  contribuer  a  l’elaboration  de  programmes  de 
formation  et  faciliter  le  transfert  de  la  formation  dans  le  milieu  de  travail. 

Etude  portant  les  pratiques  et  les  besoins  courant  en  matiere  de  gestion  des  connaissances  :  Des 
recherches  plus  approfondies  sur  ces  sujets  pourraient  aider  la  communaute  a  regler  les  questions 
bees  a  la  conservation  et  au  transfert  des  connaissances  decoulant  d’enjeux  organisationnels 
comme  le  roulement  et  le  manque  de  personnel. 

Recherche  de  nouvelles  methodes  d’ evaluation  des  produits  et  du  rendement :  En  raison  du 
manque  de  criteres  devaluation  non  ambigus  et  de  la  variabilite  des  normes  et  des  exigences  en 
matiere  d’analyse,  l’evaluation  des  produits  du  renseignement  repose  principalement  sur  le 
jugement  subjectif  des  gestionnaires.  La  communaute  tirerait  avantage  de  la  definition  de 
methodes  devaluation  du  rendement  des  analystes  plus  objectives,  par  exemple  pour  mesurer 
l’exactitude  de  l’ensemble  de  leurs  decisions. 

Analyse  organisationnelle  :  Une  analyse  organisationnelle  pourrait  contribuer  a  1’ identification 
des  sources  de  problemes,  comme  le  manque  de  personnel,  les  contraintes  de  temps,  le  roulement 
du  personnel  (surtout  les  analystes  militaires),  le  manque  de  retroaction  sur  les  produits  finaux  du 
renseignement  et  les  obstacles  dans  le  partage  de  1’ information  entre  les  differents  organismes  du 
renseignement  et  ministeres.  Tous  ces  facteurs  peuvent  avoir  une  incidence  sur  la  productivite  des 
analystes  et  contribuer  a  la  perturbation  des  processus  operationnels,  ainsi  qu’a  la  perte 
d’expertise  et  de  memoire  organisationnelle. 

Signification:  Le  present  rapport  contient  la  synthese  des  resultats  d’une  etude  unique  basee  sur 
des  entrevues  menee  aupres  de  gestionnaires  canadiens  d’ analystes  du  renseignement.  Les  divers 
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problemes  souleves  par  les  gestionnaires  sont  abordes  sur  la  base  d’etudes  sur  le  renseignement  et 
de  publications  sur  les  sciences  du  comportement.  Le  rapport  contient  l’aperqu  de  la  feuille  de 
route  d’un  programme  de  recherche  en  sciences  du  comportement  a  l’appui  de  la  fonction  du 
renseignement,  etablie  en  fonction  des  commentaires  formules  par  les  gestionnaires  du 
renseignement.  La  recherche  scientifique  dans  les  domaines  identifies  permettrait  d’ameliorer 
notre  comprehension  des  problemes  souleves  et  pourrait  fournir  des  renseignements  utiles  sur  les 
faqons  d’ameliorer  le  rendement  humain  en  matiere  de  production  du  renseignement. 

Perspectives:  Les  domaines  identifies  pour  la  recherche  et  le  developpement  englobent  un  grand 
nombre  de  problemes  et  couvrent  plusieurs  disciplines.  Les  efforts  de  recherche  dans  plusieurs  de 
ces  domaines  sont  deja  en  cours  au  TRIG,  comme  la  calibration  de  l’exactitude  du  jugement  des 
analystes,  la  communication  des  estimations  relatives  a  la  probabilite  linguistique,  le  role  de  la 
fiabilite  et  de  la  diagnosticite  de  1’ information  dans  la  prise  de  decisions,  la  conservation  de  la 
coherence  en  fonction  de  nouvelles  donnees  probantes,  l’evaluation  de  l’efficacite  de  la  formation 
sur  les  differences  culturelles,  la  relation  entre  les  differences  individuelles  et  l’exactitude  et  la 
coherence  dans  la  prise  de  decisions,  les  repercussions  de  la  formulation  des  questions  sur 
l’estimation  des  probabilites,  et  les  moyens  pour  visualiser  les  connaissances  et  1’ information. 
Toutefois,  il  serait  impossible  de  regler  toutes  les  questions  exposees  dans  ce  rapport  dans  le 
cadre  d’un  seul  projet.  L’equipe  de  recherche  etablira  les  priorites  des  futurs  travaux  par  le  biais 
d’une  collaboration  continue  et  etroite  avec  des  membres  de  la  communaute  canadienne  du 
renseignement  pour  veiller  a  ces  que  les  efforts  de  recherche  repondent  aux  besoins  de  cette 
communaute,  en  particulier  ceux  de  nos  parrains  et  de  nos  partenaires. 
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Introduction 


Intelligence  analysis  is  an  important  state  activity  aimed  to  inform  and  support  policy  and 
command  decision  making  (Davis  2006,  Jervis  1991).  The  ultimate  goal  of  the  intelligence 
function  is  to  provide  timely  and  relevant  information  to  decision  makers  to  aid  their 
understanding  of  the  issues  at  hand  and  to  allow  them  to  make  more  informed  decisions. 

Production  of  intelligence  involves  a  variety  of  activities  such  as  assessment  of  intelligence 
requirements,  search,  collection,  evaluation  and  analysis  of  information,  and  communication  of 
the  outcomes  of  the  assessments.  A  number  of  individuals  and  organizations  carry  out  these 
activities.  Intelligence  analysis  is  one  of  the  functions  in  the  myriad  of  steps  involved  in 
producing  usable  intelligence.  Intelligence  analysts  search  through,  evaluate,  select,  and  interpret 
available  information  to  produce  intelligence  products.  Intelligence  analysis,  often  characterized 
as  putting  together  a  puzzle  with  many  missing  pieces  (Johnson  2007),  is  an  inherently 
challenging  process,  characterized  by  a  great  deal  of  uncertainty  (Davis  1992,  Heuer  1999, 
Lefebvre  2004)  and  constantly  increasing  data  overload  due  to  advancements  in  information  and 
communication  technologies  (Johnson  2007,  Treverton  2001,  Woods  et  al.  2002). 

More  recent  intelligence  failures  such  as  those  that  led  to  the  tragic  events  of  September  1 1,  2001, 
and  inaccurate  assessments  of  Iraqi  weapons  of  mass  destruction  (WMD)  capability  were 
attributed  to  failures  in  analysis  (Bruce  and  George  2008).  These  events  drew  a  lot  of  public 
attention  to  the  intelligence  community  (IC).  As  a  result,  the  intelligence  communities  and 
practices  of  Canada’s  allies  have  been  subjected  to  commissions  of  inquiry  and  reviews  of 
analytic  capability  (Butler  et  al.  2004,  9/1 1  Commission  2004).  Scrutiny  of  the  IC  and  its 
practices  has  uncovered  a  number  of  potential  causes  of  intelligence  failures  including  ineffective 
leadership,  lack  of  inter-organizational  coordination  and  information  sharing  (Hulnick  2008), 
poor  quality  of  available  information  (Pritchard  and  Goodman  2009),  “lack  of  analytical 
imagination,”  that  is,  an  inability  to  generate  (unlikely)  hypotheses  which,  in  turn,  leads  to  a 
failure  to  generate  (proper)  collection  requirements  (Bruce  2008),  failures  to  properly  interpret 
available  information  due  to  cognitive  biases  and  mindsets  (Butterfield  1993,  Heuer  1999),  and 
failures  of  decision  makers  in  heeding  accurate  intelligence  assessments.  In  addition,  intelligence 
misjudgements  are  inevitable  because  the  possibility  of  a  mistake  is  inherent  in  the  nature  of 
intelligence  activity  (Brady  1993,  Heuer  1999). 

A  key  mandate  of  the  Thinking,  Risk,  and  Intelligence  Group  (TRIG)  in  the  Adversarial  Intent 
Section  (AIS)  at  Defence  R&D  Canada  -  Toronto  (DRDC  Toronto)  is  to  conduct  scientific 
activities  in  support  of  the  Canadian  intelligence  analysis  function.  Under  the  direction  of  the 
senior  author,  Dr.  David  Mandel,  and  under  the  sponsorship  of  Capt(N)  M.J.  Barber,  Director  of 
Intelligence  Capability,  Chief  of  Defence  Intelligence  (CDI),  a  5-year  Applied  Research  Program 
(ARP)  project  (15dm),  entitled  “Understanding  and  Augmenting  Human  Capabilities  for 
Intelligence  Production”  is  currently  underway.  In  the  scoping  year  for  this  project,  a  research 
team  from  TRIG  conducted  interviews  with  managers  from  two  Canadian  intelligence 
organizations.  The  purpose  of  the  interviews  was  twofold:  to  educate  the  research  team  about  the 
organization  and  challenges  of  intelligence  production  in  the  Canadian  IC,  and  to  identify  areas 
where  scientific  research  might  prove  beneficial  for  augmenting  the  intelligence  process.  The 
topics  discussed  with  the  managers  include  the  tasks  and  challenges  that  intelligence  analysts 
face,  the  skills  and  capabilities  that  are  essential  to  producing  high  quality  intelligence 
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assessments,  the  selection  process  for  analysts,  performance  evaluation,  and  current  practices  and 
developments  in  the  IC  writ  large. 

The  main  puipose  of  this  report  is  to  summarize  the  findings  from  the  interviews  conducted  with 
managers.  The  report  is  structured  in  the  following  manner:  Section  2  describes  the  study  method, 
Section  3  provides  a  summary  and  discussion  of  the  interviews,  and  Section  4  outlines  areas  for 
further  investigation  that  might  help  to  augment  the  intelligence  analysis  function. 

This  research  was  approved  by  the  DRDC  Human  Research  Ethics  Committee  (HREC)  under 
protocol  L-638. 
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2  Method 


2.1  Participants 

Seven  managers  from  two  Canadian  intelligence  organizations  -  three  from  the  International 
Assessment  Staff  (IAS)  and  four  from  CDI  -  participated  in  the  interviews.  Interviews 
were  conducted  during  the  summer  and  fall  of  2008.  Participants’  experience  in  the 
intelligence  domain  ranged  between  6  and  21  years  (median:  11  years),  and  their 
experience  in  the  management  role  within  the  intelligence  domain  ranged  between  2  and  20  years 
(median:  8  years). 

Only  half  of  the  managers  had  intelligence  analysts  directly  reporting  to  them.  However,  the 
number  of  analysts  each  manager  had  supervised  at  any  given  time  in  his  or  her  career  ranged 
between  4  and  50  (median:  8  analysts). 

Participants  for  this  study  were  not  selected  randomly.  Rather,  managers  with  whom  the  research 
team  had  established  contact  in  the  past  were  asked  to  participate  in  the  interviews.  All  seven 
managers  that  we  approached  agreed  to  participate  in  the  study,  resulting  in  a  perfect  response 
rate.  Initial  contacts  with  the  managers  had  been  established  through  the  senior  author’s  prior 
contacts  with  the  interviewees.  All  of  the  interviewed  managers  take  an  active  role  in  the 
community  and  appreciate  the  potential  value  of  scientific  research  to  their  profession.  It  is 
unclear  whether  our  sample  is  representative  of  the  general  population  of  Canadian  intelligence 
managers  or  only  of  one  of  its  more  active  sub-groups.  One  way  to  address  this  methodological 
issue  would  be  to  expand  the  sample  and  conduct  additional  interviews,  which  is  a  prospect  under 
consideration  by  the  research  team  and  the  project  sponsor. 

2.2  Procedure 

In  the  course  of  the  study,  the  research  team  conducted  six  in-person  interviews.  The  length  of  the 
interviews  varied  between  1  and  2.5  hours  depending  on  participants’  availability  and  the  length 
of  their  responses.  Five  interviews  were  individual,  while  one  interview  was  conducted  with  a 
group  of  three  participants,  one  of  whom  agreed  to  an  individual  follow-up  interview.  The 
interviews  followed  a  semi-structured  format;  therefore,  there  was  some  degree  of  variation  in  the 
questions  posed  to  different  managers  depending  on  individuals’  experience  with  different  topics 
of  interest.  The  topics  that  were  discussed  during  the  interviews  included:  the  organization’s 
intelligence  products  and  processes;  challenges  that  analysts  face;  skills  and  capabilities  essential 
to  intelligence  analysis;  selection  criteria;  training;  analytical  tools;  the  managers’  roles  in 
intelligence  production;  and  characteristics  of  the  IC  at  large.  The  initial  list  of  interview 
questions  is  included  in  Annex  A.  Not  all  of  the  questions  from  this  list  were  discussed  with  each 
manager  due  to  the  varying  length  of  the  interviews  and  the  relevance  of  the  question  in  light  of 
the  interviewee’s  experience.  Given  participants’  approval,  the  interviews  were  audio  recorded 
whenever  possible  and  later  transcribed.  If  an  audio  recording  of  an  interview  was  not  feasible 
either  due  to  demands  of  the  facility  or  personal  preferences  of  a  participant,  then  members  of  the 
research  team  took  notes  during  the  interview.  The  notes  were  later  consolidated  and  expanded. 
Interview  transcripts  or  consolidated  notes  were  sent  to  the  participants  for  review  and  approval. 
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The  research  team  analysed  the  transcripts.  The  analysed  data  set  consisted  of  six  text  documents: 
four  full  interview  transcripts  and  two  sets  of  consolidated  notes. 

2.3  Data  and  analysis 

The  interview  transcripts  and  notes  were  analysed  with  qualitative  data  analysis  software  QSR  N6 
developed  by  QSR  International  Pty.  Ltd,  Victoria,  Australia.  Transcript  sections  from  different 
interviewees  pertaining  to  the  same  topic  were  grouped  together  and  summarized  for  each  topic. 
Although  a  number  of  different  topics  were  discussed  during  the  interviews,  the  discussions 
mainly  focused  on  analysts’  tasks  and  processes.  As  an  illustration  of  the  amount  of  attention 
devoted  to  each  of  the  topics  in  the  interviews,  Figure  1  shows  the  distribution  of  the  transcript 
text  among  the  discussed  themes.  This  representation  serves  as  only  a  crude  approximation  since 
some  of  the  themes  overlapped  and  not  all  topics  were  discussed  with  each  interviewee. 

Due  to  the  small  sample  and  the  semi-structured  nature  of  the  interviews,  it  was  difficult  to  infer 
reliable  differences  between  the  two  organizations.  Therefore,  responses  from  the  IAS  and  CDI 
managers  were  aggregated  and  only  general  summaries  pertaining  to  the  topics  are  reported  in  the 
following  section.  We  did,  however,  comment  on  distinctions  between  the  two  organizations  that 
did  emerge  from  the  data. 


Analysts'  tasks  and  processes 
Analysts'  skills,  capabilities  and 
Training  and  tools 
Organizational  issu  es 
Analysts'  challenges 
Managers'  role  and  challenges 
Intelligence  community  at  large 


0°o  10°  o  20°b  30°  o  40°  o 


Figure  1:  Percentage  of  transcribed  text  by  theme 
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3  Summary  of  the  Interviews  and  Discussion 


This  section  summarizes  interview  findings  and  discusses  them  in  light  of  the  intelligence 
analysis  literature.  The  discussion  is  organized  around  the  interview  themes  outlined  in  Figure  1. 
Issues  discussed  in  this  section  are  also  summarized  in  a  Concept  Map  in  Annex  C. 

3.1  Analysts’  tasks  and  analytic  process 

Challenges  that  analysts  experience  are  a  product  of  an  analyst’s  task  and  his  or  her  ability  to 
cope  with  situational  and  task  demands,  as  determined  by  skills,  knowledge,  and  experience. 
Some  of  these  challenges  are  inherent  in  the  nature  of  the  intelligence  analysis  activity  itself  while 
other  challenges  could  be  due  to  a  particular  organizational  environment.  In  Sections  3.1  and  3.2, 
we  summarize  our  discussions  with  managers  regarding  the  nature  of  analytic  tasks, 
processes,  products,  organizational  environment,  and  resulting  challenges.  Section  3.3  focuses  on 
analysts’  skills  that  are  essential,  according  to  the  managers,  to  performing  the  tasks  and  coping 
with  challenges. 

3.1.1  Analysts’ tasks 

Results.  All  managers  commented  that  intelligence  analysts  in  their  organizations  engage  in  a 
variety  of  activities.  While  the  specific  activities  may  differ  for  analysts  in  IAS  and  CDI,  the 
general  categories  of  tasks  are  similar.  According  to  the  managers,  analysts’  activities  include 
preparing  intelligence  reports,  providing  support  to  operations,  providing  situational  awareness 
briefings  to  senior  leaders,  liaising  and  interacting  with  the  rest  of  the  IC,  fulfilling  bureaucratic 
requirements  such  as  attending  meetings  and  briefings,  and  providing  warning  intelligence,  often 
in  the  form  of  probabilistic  judgments  of  events  occurring. 

Discussion.  As  managers  indicated,  intelligence  analysts  perform  a  number  of  activities,  of  which 
analysis  is  only  one.  Intelligence  analysis  itself  involves  a  broad  range  of  activities.  Treverton  and 
Gabbard  (2008)  described  a  pyramid  of  analytic  tasks,  which  includes  five  levels  of  processing 
information  from  “raw”  data  to  finished  products,  and  each  of  the  levels  requires  different  skills 
and  tools.  Naturally,  not  all  of  these  tasks  are  completed  by  a  single  person.  Analysts  at  IAS 
mostly  perform  strategic  intelligence  analysis  while  analysts  at  CDI  mostly  engage  in  medium-  to 
short-term  operational  analysis.  In  both  organizations,  analysts  usually  do  not  collect  or  analyse 
“raw”  data  themselves,  as  they  mostly  deal  with  information  that  was  collected,  evaluated, 
organized,  and  coded  by  others.  For  example,  an  all-source  analyst  might  use  imagery 
intelligence  data  that  was  interpreted  by  a  specially  trained  imagery  analyst,  or  they  may  use  a 
media  report  for  which  information  was  selected,  processed,  and  presented  in  a  certain  way  by  the 
producers  of  the  report  (Butler  et  al.  2004,  Pritchard  and  Goodman  2009).  Our  interest  and 
discussions  focused  mostly  on  the  issues  related  to  conducting  all-source  intelligence  assessments 
and  preparing  analytic  reports.  The  following  subsections  examine  the  analytic  processes. 
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3.1.2  The  “Intelligence  cycle” 


Results.  Conducting  intelligence  assessments  involves  a  variety  of  activities,  including:  defining 
the  topic  and  the  analytic  questions;  conducting  information  searches;  evaluating,  interpreting, 
integrating,  and  analyzing  information;  making  judgments;  and  communicating  the  final 
assessment.  The  intelligence  production  process  is  often  conceptualized  as  a  five-stage 
cycle  (see  Figure  2  for  an  example  of  the  intelligence  cycle).  However,  some  managers 
commented  that  the  actual  process  is  not  as  straightforward  as  it  appears  from  the  cyclic  model. 
As  one  interviewee  stated: 

There  is  a  process  [the  intelligence  cycle];  it  is  a  theoretical  process.  The 
implementation  of  it  is  something  completely  different  ...  .  There  is  a  big 
difference  between  what  should  be  done  and  what  they  [analysts]  are  all  taught 
to  do,  and  actually  what  they  are  able  to  do  at  any  given  time. 


Figure  2:  The  Traditional  Intelligence  Cycle,  from  Johnston  (2005) 


Nevertheless,  some  managers  see  the  intelligence  cycle  as  a  helpful  model  for  teaching 
intelligence  production  processes: 

The  intelligence  cycle  is  not  an  accurate  depiction  of  how  things  are  actually 

done,  but  it’s  a  broadly  useful  framework.  (Interviewee) 

Discussion.  The  intelligence  cycle  model  of  the  intelligence  production  process  originated  in  the 
military  context  and  was  adopted  by  the  civilian  community  (Herman  1996).  The  military 
intelligence  cycle  is  a  simplified  prescriptive  model  of  a  complex  process,  with  emphasis  on  a 
linear  sequence  of  events  and  processes  (Herman  1996,  Johnston  2005).  The  civilian  IC  does  not 
have  its  own  formal  intelligence  doctrine  and  it  adopted  the  military  intelligence  cycle  as  an 
accurate  description  of  the  process.  The  model  is  based  on  the  notion  that  the  whole  process  is 
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driven  by  user  requirements  (Herman  1996).  Although  the  cycle  idea  is  “deeply  ingrained”  in  the 
community  (Treverton  2001),  our  interviewees  echoed  some  of  the  criticisms  this  model  has 
received  in  the  literature.  These  are  described  below. 

One  challenge  to  the  model  is  its  assumption  that  the  intelligence  production  process  is  driven  by 
consumer  requirements  (Herman  1996,  Hulnick  2006,  Treverton  2001).  However,  intelligence 
consumers  may  not  always  be  able  to  articulate  their  needs.  In  some  cases,  they  may  be  aware  of 
pertinent  issues  but  lack  the  time  or  patience  to  formulate  well-structured  queries  to  the  IC.  In 
other  cases,  policy  makers  may  have  limited  awareness  of  pertinent  issues,  and,  in  such  cases,  the 
IC  must  bring  them  to  the  decision  makers’  awareness.  Thus,  even  in  instances  where  intelligence 
production  is  policy  directed,  the  intelligence  community  will  likely  play  an  important  role  in 
defining  and  refining  the  questions.  Contrary  to  the  model,  the  process  “is  driven  by  intelligence 
‘pushing’  rather  than  policy  ‘pulling’  ”  (Treverton  2001,  p.  106).  Former  National  Security 
Advisor  to  United  States  (US)  Presidents  Gerald  Ford  and  George  H.  W.  Bush,  Lt.  Gen  (Ret.) 
Brent  Scowcroft,  endorsed  this  view  in  a  reply  to  a  question  posed  to  him  at  a  recent  National 
Academy  of  Sciences  public  workshop  on  intelligence  analysis.  Scowcroft  was  asked  whether  the 
bulk  of  intelligence  was  pulled  by  policy  makers  or  pushed  by  the  IC,  to  which  he  replied  “...  I 
would  say,  predominantly,  it  is  the  flow  from  the  intelligence  community  to  the  decision  maker” 
(see  Scowcroft  2009,  for  the  link  to  the  audio  recording  of  his  comments). 

Another  critique  of  the  intelligence  cycle  model  is  that  it  oversimplifies  the  process  and  does  not 
include  all  aspects  of  the  intelligence  function,  such  as  counterintelligence  and  covert  action 
(Hulnick  2006).  All  models  are  simplifications  of  the  phenomena  they  are  intended  to  represent, 
and  although  there  is  value  to  be  gained  from  some  simplification,  too  much  can  obscure 
important  attributes  of  the  phenomena.  The  intelligence  cycle  model  captures  key  elements  of  the 
production  process,  but  does  not  articulate  in  any  detail  the  composition  of  these  main  elements. 
A  benefit  of  excluding  such  details  is  that  the  model  can  be  applied  to  many  intelligence 
organizations  and  across  the  tactical  -  strategic  continuum.  A  highly  specified  model  would  not 
apply  across  intelligence  organizations  because  processes  vary  widely  among  (and  even  within) 
organizations.  A  drawback  of  the  model’s  over- simplification  is  that  it  may  do  little  to  facilitate 
analysts’  understanding  of  their  responsibilities  and  challenges  (Johnston  2005).  The  neat  cyclical 
representation  also  has  been  criticized  for  not  realistically  representing  the  serial  and  parallel 
nature  of  the  flow  of  operations  in  intelligence  production  (Herman  1996,  Johnston  2005, 
Treverton  and  Gabbard  2008).  Stages  in  the  cycle  (e.g.,  collection  and  analysis)  might  overlap 
rather  than  being  clearly  distinguished  as  implied  by  the  model  (Treverton  and  Gabbard  2008). 

Adjustments  to  the  sequence  and  contents  of  stages  in  the  intelligence  cycle  have  been  proposed 
(Herman  1996,  Hulnick  2006,  Omand  2009,  Treverton  2001,  Treverton  and  Gabbard  2008).  For 
example,  Herman  (1996),  Treverton  (2001),  and  Hulnik  (2006)  highlighted  the  fact  that  the  real 
sequence  of  information  flow  is  more  complicated  than  what  is  depicted  in  the  traditional 
intelligence  cycle  and  suggested  sub-cycles  and  shortcuts  among  the  stages.  Further,  they 
maintain  that  some  stages  (e.g.,  collection  and  analysis)  run  parallel  to  each  other  rather  than  in  a 
sequence.  Similarly,  Omand  (2009)  represented  the  cycle  as  an  interactive  network  of  five 
components:  direction,  accessing,  elucidation,  dissemination,  and  action-on.  The  “accessing” 
stage  replaced  the  traditional  “collection”  stage,  thus  highlighting  the  evolution  of  availability  and 
access  to  information  since  the  Cold  War.  Similarly,  “elucidation”  replaced  “analysis”,  putting  an 
emphasis  on  analysts’  abilities  to  derive  meaning  from  the  information  available.  “Direction”  and 
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“dissemination”  stages  remain  in  Omand’s  model,  but  he  added  a  new  component,  “action-on”,  to 
highlight  the  significance  of  intelligence  in  short-term  security  activities. 

Even  with  the  proposed  changes  in  the  flow  and  content  of  the  cycle’s  stages,  the  adjusted 
versions  of  the  intelligence  cycle  represent  a  collection  of  five  or  so  interconnected  steps,  which 
still  fail  to  capture  and  communicate  the  underlying  complexity  of  the  real  process  (e.g.,  Johnston 
and  Johnston  2005).  Johnston  and  Johnston  (2005)  proposed  a  more  significant  elaboration  of  the 
intelligence  cycle  with  a  systems  model  of  the  process  (see  Figures  3  and  4).  This  model  of 
throughput  of  intelligence  products  incorporates  a  significant  number  of  factors  and  provides  a 
greater  level  of  detail  and  appreciation  of  the  complexity  of  the  intelligence  production  process. 
The  model  suggests  that  an  analyst’s  ability  to  produce  is  affected  by  a  variety  of  factors.  Some 
factors  are  internal  to  the  analyst  (e.g.,  capabilities),  some  are  properties  of  the  product  (e.g., 
complexity  of  a  document),  and  some  are  characteristics  of  the  task  setting  (e.g.,  political  and 
cultural  values  of  an  organization).  Despite  its  elaboration  on  the  traditional  model,  Johnston  and 
Johnston’s  model  does  not  identify  the  types  of  personnel  involved  in  the  process  and  so  it  is 
unclear  who  is  responsible  for  each  task  and  process.  Further,  it  assumes  that  the  process  begins 
with  consumer  requirements  and  follows  a  predefined  sequence;  these  concerns  mirror  those  of 
the  traditional  cycle  discussed  above. 


The  Components  of  the  Systems  Model 


Icon 

Purpose 

Stock 

Stocks  represent  accumulations.  These  are  quantities 
that  can  increase  or  decrease,  such  as  the  amount  of 
work  that  needs  to  be  completed,  the  time  available  in 
which  to  do  it,  experience  one  might  bring  to  a  task. 

Flow 

Flows  represent  activities.  1'hey  control  the  filling  or 
draining  of  stocks,  causing  conditions  to  change. 
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Converter 

Converters  charge  inputs  into  outputs.  They  usually 
represent  the  variables  that  initiate  change.  In  the 
example,  a  converter  might  represent  a  sudden  and 
drastic  world  event. 
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Figure  3:The  components  of  the  systems  model  of  the  intelligence  cycle  (Johnston  and  Johnston 

2005) 


DRDC  Toronto  TR  2010-144 


Volume  of  requests 
for  Intell  Products 


Demand  on  Analysts 


Completed  Intell 
Products 


Increase  or  Decrease  in 
requirements 


Change  in  demand 

(3  l  T> 

Complexity  of  Document  i  Turnaround  time 


Change  in  Analysts 
Ability  to  Produce 


6  /  \, 

zd\  and  l 


b 


Product  Evaluation 


Level  of  need 


Requirements  f  I 

a  l 

National  events 


6 

World  events 


Political  and  L  |  Process 

Type  of  Document  cultural  Values  of  \  Analysts  Capabilities 

Organization  $ 


Collected  data 


Available  data 


Change  in  the  amount  Effects  of  Requirements 
of  Collected  data 


Research 

Demands 


.  Change  in 

Relevant  Usable  Data  [  Capabilities 


Relevant  data 

Evaluation  of 
L  Relevance 


Training 


Experience 


Formatting  Processing 
of  Available  Data 


Additional  or  Revised  T  ,  „  . 

_  _  iLevel  or  Relevance 

Customer  Requests  '  f  p  d  t 


Relevance  of 
original  request 


New  issues 
raised 


Figure  4:  The  systems  model  of  the  intelligence  cycle  according  to  Johnston  and  Johnston  (2005) 


As  indicated  earlier,  models  serve  to  simplify  phenomena  and,  thus,  inevitably  omit  some  aspects 
and  highlight  others.  Appropriate  models  may  be  useful  frameworks  for  discussion  and  for 
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highlighting  important  aspects  of  underlying  phenomena.  However,  it  is  the  context  and  purpose 
of  the  model’s  application  that  determines  which  aspects  are  important.  Different  models  of  the 
same  phenomenon  may  be  created  in  order  to  draw  attention  to  different  features.  For  example,  it 
may  be  useful  to  provide  students  in  an  introductory  intelligence  analysis  course  with  a  simple 
model  that  represents  the  intelligence  process  (e.g.,  the  traditional  intelligence  cycle  with  some 
modifications)  but  the  same  model  may  be  ineffective  for  discussing  contingency  variables  in  the 
process  (for  which  a  systems  model  may  be  more  appropriate).  Any  model  of  the  intelligence 
process  is  bound  to  be  incomplete;  the  question  is  whether  or  not  a  given  simplified 
representation  provides  an  advantage  to  the  analyst.  Johnston  and  Johnston  (2005)  accurately 
state  that  “the  traditional  intelligence  cycle  model  should  either  be  redesigned  to  depict 
accurately  the  intended  goal,  or  care  should  be  taken  to  discuss  explicitly  its  limitations  whenever 
it  is  used”  (p.55).  We  would  add  that  revisions  to  the  model  should  depend  on  the  puipose  for 
which  it  is  intended. 

3.1.3  Uncertainty 

Results.  Uncertainty  is  one  of  the  inherent  characteristics  of  intelligence  analysis. 

The  problems  the  analysts  are  dealing  with  are  usually  not  structured.  There  is  a 
lot  of  uncertainty...  .(Interviewee) 

In  our  discussions,  some  managers  identified  degree  of  uncertainty  associated  with  a  particular 
analytic  topic  as  one  of  the  major  determinants  of  difficulty  in  preparing  an  intelligence  product. 
An  ability  to  cope  with  uncertainty  was  also  identified  as  one  of  the  essential  skills  required  of 
intelligence  analysts. 

Discussion.  Conceptually,  intelligence  analysis  is  the  business  of  reducing  uncertainty  based  on 
available  information  and  reasoning.  Thus,  uncertainty  is  one  of  its  fundamental  characteristics 
(Davis  1992,  Heuer  1999,  Lefebvre  2004).  In  a  somewhat  simplistic  manner,  the  analytic  process 
can  be  conceptualized  as  an  open  system  (Katz  and  Kahn  1978)  with  inputs,  a  transformation 
process,  and  outputs  ( Figure  5). 


Figure  5:  Systems  view  of  the  analytic  process 

In  this  system,  collected  information,  analysts’  knowledge,  product  requirements,  and  feedback 
from  the  environment  (i.e.,  consumers)  constitute  inputs',  manipulation  of  the  inputs  or  the 
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process  of  analysis  and  product  development  constitute  the  transformation  process',  and  the 
resulting  analytic  product  and  judgments  constitute  the  outputs. 

In  intelligence  analysis,  uncertainty  is  present  in  all  three  of  these  element-blocks  of  the  system. 

•  Inputs'.  Several  sources  of  uncertainty  exist  at  the  level  of  inputs: 

♦  As  quantity  of  available  information  increases,  the  difficulty  in  detecting  signal 
within  noise  also  increases. 

♦  Analysts  do  not  always  have  the  means  to  determine  whether  or  not  they  have 
sufficient  information  to  terminate  their  searches,  and  are  often  uncertain  regarding 
the  relative  value  of  gathering  additional  information. 

♦  Due  to  gaps  in  available  information,  analysts  have  to  supplement  missing 
information  with  judgments. 

♦  Inaccuracy  of  information,  which  may  be  due  to  deliberate  deception  or 
misinterpretation  of  data  during  processing. 

♦  Product  requirements  (i.e.,  analytic  questions  to  be  answered  by  a  report)  are  often 
ambiguous,  and  they  also  constitute  inputs  to  the  system. 

♦  Feedback  from  the  environment  is  an  input  to  the  system  that  is  crucial  for  the 
system’s  ability  to  adjust.  Some  managers  commented  that  there  is  a  lack  of 
feedback  from  consumers  regarding  the  impact  of  intelligence  products,  which 
contributes  to  uncertainty  about  product  requirements. 

♦  Stringent  time  constraints,  which  limit  the  analyst’s  capacity  to  thoroughly  search 
for  and  collect  information,  might  also  increase  input  uncertainty. 

•  Transformation  process:  The  process  of  analysis  involves  making  sense  of  information  and 
drawing  inferences.  The  uncertainty  of  the  transformation  process  in  intelligence  analysis  is 
due  in  part  to  the  uncertainty  associated  with  quality  and  completeness  of  the  inputs  (i.e., 
“garbage  in,  garbage  out”);  and  it  is  also  the  case  because  analysis  relies  largely  on  human 
reasoning  and  decision  making.  The  latter  -  although  superior  to  artificial  intelligence  in 
dealing  with  uncertainty  -  is  still  far  from  being  a  flawless  process.  Decades  of  research  in 
the  area  of  judgment  and  decision  making  have  shown  that  human  reasoning  naturally  relies 
on  cognitive  heuristics,  and  is  susceptible  to  bias  and  error  (e.g.,  Gilovich  et  al.  2002,  Heuer 
1999,  Kahneman  et  al.  1982).  Davis  (1992)  pointed  out  that  complexity,  uncertainty,  and 
time  pressure  of  intelligence  analysis  increase  analysts’  reliance  on  mindsets,  defined  as 
“distillation  of  the  intelligence  analyst’s  cumulative  factual  and  conceptual  knowledge  into  a 
framework  for  making  estimative  judgments  on  a  complex  subject”  (p.33). 

•  Outputs :  Outputs  of  intelligence  analysis  are  generally  judgments  with  respect  to  future 
states  or  events.  “ Intelligence  analysis  is  a  business  of  forecasting,  predicting  the  future  ... 

.  ”  (interviewee),  and  a  certain  degree  of  uncertainty  and  possible  misjudgements  are 
inherent  in  intelligence  assessments  (Brady  1993,  Heuer  1999). 

In  short,  uncertainty  will  always  be  a  fundamental  feature  of  the  intelligence  process,  and  efforts 
to  manage  it  more  effectively  ought  to  be  sought.  Despite  the  intuitive  appeal  of  collecting 
additional  information  to  resolve  uncertainty,  this  strategy  may  not  be  effective  or  feasible: 
necessary  information  may  not  exist,  the  information  may  be  unattainable  or  unreliable,  or 
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requirements  for  search  and  collection  may  exceed  the  time  available.  Lefebvre  (2004)  suggested 
that  uncertainty  "...  can  be  reduced  with  the  use  of  logic,  relevant  methodologies,  analytic 
techniques,  and  better  collection”  (p.248).  Some  scholars  have  suggested  using  analytical 
imagination  (i.e.,  envisioning  alternative  hypotheses  or  explanations)  to  compensate  for 
informational  gaps  and  better  direct  search  and  collection  efforts  to  reduce  uncertainty  (Bruce 
2008,  Pritchard  and  Goodman  2009).  Although  imagination  may  reduce  uncertainty  in  some 
instances,  generating  additional  plausible  explanations  might  cause  additional  uncertainty 
especially  when  the  available  evidence  does  not  allow  rejecting  alternative  explanations. 

3.1.4  Deciding  what  to  analyse  (topic  selection  for  analytic  products) 

Results.  The  main  outputs  of  intelligence  analysts’  activities  are  intelligence  assessments  in  the 
form  of  written  reports  (e.g.,  summaries  and  memoranda)  and  briefings  to  military  commanding 
officers  and  government  officials.  There  are  several  types  of  reports  produced,  which  vary  in  their 
focus,  length,  intended  audience,  publication  frequency,  time  horizon,  and  complexity.  As  noted 
earlier,  analytic  products  produced  by  the  two  organizations  we  sampled  -  namely,  CDI  and  IAS 
-  differ  in  scope  and  time  horizon. 

Managers’  opinions  varied  with  respect  to  the  main  goal  of  analytic  reports  -  some  managers  saw 
the  main  goal  of  the  reports  as  making  judgments  and  predictions,  while  other  managers  saw  the 
main  goal  of  the  reports  as  providing  situational  awareness  but  not  predictions.  However,  despite 
this  difference  in  emphasis,  managers  agreed  that  analysts  have  to  make  predictions  in  most 
analytic  assessments. 

Before  analysts  can  begin  working  on  generating  an  intelligence  product,  however,  a  certain  topic 
or  set  of  intelligence  questions  for  that  document  have  to  be  identified.  The  intelligence  cycle 
indicates  that  requirements  are  generated  by  the  consumers.  However,  most  managers  indicated 
that  both  managers  and  analysts  take  an  active  role  in  setting  specific  requirements  for  an 
assessment.  Managers  from  both  organizations  indicated  that  general  directions  for  analytic 
products  are  set  by  senior  leaders,  but  the  specific  foci  of  the  reports  are  negotiated  among  an 
analyst,  his  or  her  manager,  and,  sometimes,  the  client. 

Very  rarely  do  we  get  very  specific  questions  from  clients,  so  what  we  have  to  do 
is  try  and  second  guess  but  be  aware  of  what  the  policy  concerns  are  among  our 
clients  so  that  we  can  identify  those  questions.  ...  It’s  an  interactive  process 
between  me  and  my  analysts  in  setting  up  priorities.  ...  We  have  to  keep  focused 
on  our  clients,  but  the  clients’  needs  are  broader  than  the  questions  they  articulate 
today.  ...  There  is  no  formed  process  in  identifying  the  issues  but  I  encourage  my 
analysts  to  be  in  close  contact  with  the  clients.  Sometimes  there  is  a  negotiation 
about  a  question  because  the  question  that  the  client  is  asking  is  not  really  the 
question  they  are  really  interested  in.  This  is  what  they  think  they  are  interested  in 
and  it  could  be  something  we  cannot  do  much  with,  but  a  slightly  different  version 
will  give  us  sufficient  flexibility.  It  is  often  a  negotiation  process  and  we  work  it 
out  with  the  client.  (Interviewee) 

Managers  play  an  important  role  in  setting  the  foci  for  the  intelligence  products  and  in 
formulating  analytic  questions,  but  analysts  also  have  input  into  this  process.  Some  managers 
commented  that  IAS  analysts  have  more  flexibility  in  topic  selection  than  CDI  analysts,  perhaps 
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due  to  the  difference  in  scope  of  the  assessments  between  the  organizations  -  IAS  focuses  on 
long-term  strategic  analysis,  and  CDI  focuses  on  more  short-term  operational  analysis,  the  latter 
of  which  may  have  more  specific  foci  at  the  outset.  In  addition,  IAS  analysts  usually  begin 
developing  topics  for  potential  products  before  they  are  officially  assigned  to  producing  a 
particular  report,  while  CDI  analysts  mostly  have  to  address  their  clients’  immediate  needs. 

Discussion.  Given  that  the  main  goal  of  intelligence  assessments  is  to  provide  a  better 
understanding  for  decision  makers  (Herman  1996,  Treverton  2001),  it  is  not  surprising  that  two 
key  functional  aids  to  decision  making  were  emphasized  in  the  interviews  -  improving  situational 
awareness  and  offering  predictive  judgments.  These  two  functions  roughly  correspond  to 
different  temporal  foci,  both  of  which  may  play  a  key  role  in  improving  decision  making. 
Situational  awareness  provides  the  decision  maker  with  knowledge  about  the  state  of  the  world  in 
the  area  of  interest  in  the  recent  past  and  the  present.  Of  course,  the  difference  in  the  way 
managers  prioritize  intelligence  objectives  may  also  reflect  differences  in  the  requirements  with 
which  their  respective  organizations  or  offices  are  tasked. 

Because  analysts  produce  intelligence  to  support  policy  makers  and  commanding  officers,  it  is 
crucial  that  their  products  are  relevant  to,  and  satisfy,  their  consumers’  needs  (Herman  1996, 
Treverton  2008).  Selection  of  intelligence  questions  to  be  answered  by  an  intelligence  product  is 
closely  related  to  the  analyst’s  (and  manager’s)  ability  to  identify  consumers’  needs.  Ideally, 
intelligence  consumers  would  clearly  specify  their  intelligence  requirements.  However,  as  some 
managers  have  pointed  out,  and  as  discussed  in  the  previous  section,  this  is  rarely  the  case. 
Intelligence  consumers  are  not  always  able  to  clearly  specify  their  intelligence  requirements,  or 
when  the  consumers  are  at  last  able  to  articulate  their  requirements,  there  may  not  be  enough  time 
left  for  collection  and  analysis  to  answer  consumer’s  questions.  In  addition,  consumers  do  not 
always  have  a  good  understanding  of  intelligence  capabilities  and  what  intelligence  analysts  can 
and  cannot  do  (Herman  1996).  This  disconnect  creates  on  the  one  hand,  the  need  for  intelligence 
producers  to  anticipate  what  consumers  may  want  in  the  future  and  develop  their  collection  and 
understanding  accordingly  (Hulnick  2006,  Treverton  2001);  and,  on  the  other  hand,  it  creates  the 
need  for  negotiation  with  consumers  in  an  attempt  to  arrive  at  a  common  understanding  of  the 
requirements  and  realistic  expectations.  Indeed,  "...  there  must  be  good  communication  between 
policy  consumers  and  intelligence  managers  if  intelligence  is  to  be  on  target  and  meet  the  needs 
of  decision  makers”  (Hulnick  2006,  p.968). 

Although  it  was  not  raised  by  managers  that  we  interviewed,  in  the  complex  consumer  -  producer 
relationship  there  is  always  a  risk  of  consumers  pressuring  intelligence  personnel  to  deliver 
products  that  conform  to  consumers’  pre-conceived  views  (Gardiner  2009,  Hulnick  2006).  An 
example  of  intelligence  yielding  to  such  pressure  are  the  infamous  estimates  on  WMD  that  were 
used  as  justification  by  the  administration  of  US  President  George  W.  Bush  to  invade  Iraq  in  2003 
(Hulnick  2006,  Treverton  2008),  although  the  role  that  this  pressure  played  among  other  factors 
in  the  resulting  intelligence  failure  is  still  debated  (Jervis  2009).  Many  writers  agree  that 
intelligence  producers  need  to  be  in  a  closer  relationship  with  consumers  to  ensure  that  products 
are  relevant  and  that  they  address  consumers’  needs  more  effectively  (Davis  1995,  Gardiner  2009, 
Medina  2009).  However,  the  IC  is  faced  with  the  dilemma  of  how  to  bring  analysts  closer  to 
consumers  while  avoiding  politicization1  of  intelligence  (Gardiner  2009,  Treverton  2008). 


1  Politicization  of  intelligence  refers  to  distortion  of  analysis  resulting  from  pressure  or  willingness  to 
satisfy  demands  of  intelligence  managers  or  consumers.  Treverton  (2008)  differentiates  among  five  forms 
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3.1.5  Assessment’s  difficulty 


Results.  There  are  a  variety  of  intelligence  products  that  intelligence  organizations  create,  and 
even  products  of  the  same  type  (e.g.,  intelligence  memoranda)  might  vary  greatly  in  their  level  of 
difficulty.  The  main  factors  mentioned  by  managers  that  contribute  to  an  assessment’s  difficulty 
include: 

•  The  topic:  The  nature  of  the  issue  being  analysed  and  the  number  of  aspects  that  need  to  be 
considered  affect  difficulty.  For  example,  analysing  an  issue  for  a  region  that  consists  of 
several  countries  is  usually  more  difficult  than  analysing  the  same  issue  for  a  single  country. 

•  Time  horizon:  The  more  forward-looking  the  report,  the  more  uncertainty  an  analyst 
has  to  manage. 

•  Amount  and  quality  of  available  information:  Large  quantities  of  information,  especially 
when  coupled  with  low  evidential  quality  (e.g.,  low  source  credibility  or  information 
reliability)  contribute  to  the  difficulty  of  an  assessment.  It  requires  more  time  and  effort  to 
search  through  large  quantities  of  available  information  in  order  to  find  that  which  is  not 
only  relevant  but  is  also  reliable.  It  may  be  practically  impossible  to  review  everything  that 
is  relevant  to  a  topic  in  a  limited  amount  of  time,  which  contributes  to  analysts’  uncertainty 
regarding  the  potential  existence  of  other  useful  information  that  was  missed  in  the  initial 
search.  To  cope  with  this  uncertainty,  some  analysts  may  prolong  their  information¬ 
gathering  at  the  expense  of  analysis.  Some  participants  also  noted  that  a  large  quantity  of 
available  information  does  not  ensure  sufficient  reliable  information  for  testing 
considered  hypotheses.  As  a  result,  analysts  may  feel  pressure  to  fill  information  gaps  with 
analyses  and  judgment. 

•  Available  time:  Stringent  timelines  increase  the  difficulty  of  assessments. 

•  Background  knowledge:  If  analysts  lack  background  knowledge  on  an  issue  being  analysed, 
the  difficulty  of  the  task  increases. 

•  The  level  of  analytical  assessment  required  for  the  report:  Simply  summarising  information 
coming  from  different  sources  is  easier  than  performing  a  deep  analysis  of  an  issue  or  trend, 
and  providing  an  explanation  or  judgment. 

Discussion.  Managers  commented  on  several  sources  of  difficulty  in  making  intelligence 
assessments,  which  may  be  organized  into  three  general  groups  of  factors: 

•  individual  characteristics  of  analysts  (e.g.,  their  background  knowledge), 

•  inherent  properties  of  the  task  (e.g.,  the  specific  issue  being  analysed),  and 

•  the  environment  in  which  the  assessment  is  carried  out  (e.g.,  the  amount  and  quality  of 
information  and  time  constraints). 


of  politicization:  “Direct  pressure”  from  policy  officials;  a  “House  line”  -  pressure  coming  from  analytic 
office;  “Cherry  picking”  -  policy  makers  pick  “convenient”  assessments  among  those  available;  “Question 
asking”  -  framing  of  the  question  affects  the  answer;  and  “Shared  mindset”  -  where  intelligence  and  policy 
share  strong  presumptions.  For  a  detailed  discussion  see  Treverton  (2008) 
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One  way  of  summarising  the  points  raised  by  managers  is  to  say  that  the  difficulty  of  a  task  is  a 
function  of  the  analyst’s  individual  characteristics  and  the  environment  in  which  that  task  is 
performed.  We  would  add  that  the  interactions  between  person  and  environmental  factors  are  also 
central  to  understanding  objective  and  perceived  task  difficulty.  Indeed,  that  most  of  human 
behaviour  is  the  product  of  person  x  environment  interactions  is  one  of  the  foundational  insights 
of  social  psychology  (Brunswik  1943,  Heider  1958,  Lewin  1935).  The  analyst’s  environment,  in 
this  sense,  includes  the  task  itself,  which  has  a  certain  degree  of  complexity,  and  the  setting  in 
which  the  task  is  performed,  which  includes  available  resources,  processes,  expectations  and 
relationships  with  other  individuals.  The  analyst’s  motivation,  skills,  knowledge  and  experience 
influence  how  the  task  is  perceived  in  a  given  context.  For  a  discussion  of  an  extended  set  of 
variables  that  play  a  role  in  intelligence  production,  see  Johnston  (2003,  2005). 

All  three  aspects  -  inherent  properties  of  the  task,  task  setting,  and  individual  characteristics  of 
analysts  -  have  been  discussed  in  the  intelligence  literature.  For  instance,  inherent  properties  of 
analytic  tasks  are  used  to  classify  intelligence  problems  into  puzzles,  mysteries  (Treverton  2001) 
and  messes  (Thompson  2010).  Puzzles  are  intelligence  questions  that  have  clear  boundaries  and  a 
definitive  answer  given  the  availability  of  certain  reliable  information.  Mysteries,  while  still 
having  clear  boundaries,  do  not  have  a  definitive  answer  regardless  of  the  information  available. 
Messes  have  neither  correct  solutions  nor  clear  boundaries,  making  it  difficult  to  define  a  solution 
state  (Thompson  2010).  The  emphasis  in  this  classification  is  placed  on  task  complexity, 
which  is  determined  by  the  existence  of  the  “correct”  answer  and  the  availability  of  resources 
(e.g.,  information).  Resource  availability  is  part  of  the  setting  in  which  the  task  is  performed. 
Jones  (1998)  and  Krizan  (1999)  described  a  more  elaborate  taxonomy  of  intelligence  problems 
along  the  dimension  of  uncertainty.  They  differentiate  among  five  problem  types:  simplistic, 
deterministic,  moderately  random,  severely  random,  and  indeterminate.  Each  problem  type  is 
further  described  along  a  set  of  characteristics  such  as  role  of  facts,  role  of  judgment,  analytical 
task,  analytical  method,  and  probability  of  error.  For  example,  the  role  of  facts  for  the  five 
problem  types  decreases  as  we  move  from  simplistic  to  deterministic  to  moderately  random  and 
so  forth,  and  the  role  of  judgment  and  probability  of  error  increases  (see  Annex  B  for  the 
complete  taxonomy).  Assessments  that  analysts  at  CDI  and  IAS  perform  most  of  the  time  seem  to 
be  intelligence  mysteries  (or,  perhaps,  even  messes)  rather  than  puzzles  that  fall  within  the  range 
of  “moderately  random”  to  “indeterminate”  based  on  Krizan’ s  taxonomy.  Problems  characterized 
by  high  uncertainty  rely  more  on  human  judgment,  and  are  associated  with  a  higher  probability  of 
error  (than  problems  with  low  uncertainty),  and  are  concerned  mainly  with  predicting  future 
events  and  situations.  Although  Krizan  did  not  make  the  mapping  between  this  taxonomy  and 
task  difficulty,  it  is  reasonable  to  associate  the  uncertainty  of  a  task  with  its  difficulty.  Krizan 
suggested  that  her  model  might  enable  decision  makers  and  analysts  to  articulate  a  particular 
scenario  and  to  assess  the  capabilities  needed  to  solve  the  problem.  The  model  offers  a  structured 
approach  to  classifying  and  characterising  intelligence  problems  based  mostly  on  properties  of  the 
task.  Although  the  model  may  be  a  useful  theoretical  approach  to  differentiate  among  intelligence 
problems,  its  practical  utility  remains  to  be  determined. 

A  number  of  issues  related  to  the  task  setting  have  been  discussed  (Johnston  2005,  Pritchard  and 
Goodman  2009,  Treverton  2001).  For  instance,  Hedley  (2007)  and  Treverton  (2001)  noted  that 
the  constantly  increasing  amount  of  available  information  creates  data  overload  for  analysts  and 
blurs  the  distinction  between  analysis  and  collection.  Some  practitioners  have  observed  an  overall 
trend  of  increasing  scope  and  complexity  of  intelligence  issues  in  the  post  Cold  War  era 
(McLaughlin  2008).  Treverton  (2001)  further  observed  that  an  increase  in  the  number  of  possible 
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security  concerns  implies  that  analysts  may  be  faced  with  a  greater  variety  of  topics  and,  as  a 
result,  may  have  limited  opportunity  to  specialize.  Heightened  public  attention  to  the  intelligence 
function  after  the  terrorist  suicide  attacks  on  the  US  on  1 1  September  2001  (9/1 1  attacks),  and  the 
Iraqi  WMD  failures  puts  intelligence  analysis  in  the  spotlight  of  public  and  political  attention, 
which  creates  additional  pressure  for  analysts  (Bruce  and  George  2008,  Lowenthal  2008).  In 
addition,  organizational  product  evaluation  processes  may  impose  additional  time  limitations  on 
analysts  (Johnston  2005). 

Discussion  of  the  professionalization  of  intelligence  analysis  has  sparked  interest  in  the  individual 
characteristics  -  abilities,  skills,  and  knowledge  -  required  of  analysts  to  effectively  carry  out 
their  tasks  (Krizan  1999,  Moore  and  Krizan  2003).  However,  discussion  of  environmental  and 
individual  factors  in  intelligence  analysis  has  mostly  been  fragmented  and  focused  on  specific 
areas  of  interest.  A  unifying  view  of  all  of  the  issues  involved  has  not  been  provided  with  the 
exception  of  the  systems  model  of  Johnston  and  Johnston  (2005)  of  the  intelligence  cycle 
discussed  in  Sub-section  3.1.2. 

3.1.6  Information  dependence 

Conducting  intelligence  analysis  is  like  trying  to  solve  a  jig-saw  puzzle  without 
having  the  final  picture  and  no  certainty  that  you  have  all  the  pieces. 
(Interviewee) 

The  jigsaw  puzzle  analogy  of  intelligence  analysis  offered  by  several  of  our  interviewees,  and 
used  in  the  literature  to  describe  intelligence  (e.g.,  Johnson  2007),  highlights  the  dependency  of 
analytic  activities  on  the  information  available.  This  analogy  implies  that  analysts  have  to 
reconstruct  a  comprehensive  picture  of  a  situation  from  segregated  pieces  of  information. 
Analysts  deal  with  a  dynamic  world  and  their  understanding  of  the  unfolding  events  relies  heavily 
on  the  information  available  (Bruce  2008).  Information  processing  is  one  of  the  predominant 
activities  in  intelligence  analysis.  It  includes  information  search,  interpretation,  evaluation,  and 
selection  (i.e.,  of  relevant  information  for  a  particular  analytic  problem).  Analysts  rely  on 
different  sources  of  information  and,  most  of  the  time  (if  not  always),  have  to  sift  through  a 
massive  amount  of  available  data  to  extract  relevant  and  useful  information  for  their  purposes 
(Hedley  2007,  Johnston  2005,  Pritchard  and  Goodman  2009,  Treverton  2001,  Woods  et  al.  2002). 
In  the  following  subsections,  we  discuss  managers’  comments  regarding  the  sources  of 
information  available  to  analysts,  the  process  of  information  evaluation,  and  the  role  that 
classification  level  plays  in  information  selection. 

3.1 .6.1  Information  sources 

Results.  Analysts  in  CDI  and  IAS  do  not  collect  “raw”  intelligence  themselves,  but  draw  on  open 
and  classified  information  available  from  various  sources  (e.g.,  open-source  information  from  the 
Internet  and  other  media,  classified  analyzed  imagery  data,  collected  human  intelligence,  etc.). 
Analysts  obtain  their  information  mainly  from  computer  information  systems  and  social  networks 
of  experts  and  peers.  Broadly  speaking,  the  sources  of  information  that  analysts  use  are: 

•  Open  source,  publicly  available,  information  systems  such  as  the  Internet,  newspapers, 
television,  and  radio; 
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•  Specialized  computer  information  systems  of  varying  classification  levels,  which  contain 
classified  information  from  various  sources  such  as  human  intelligence  (HUMINT),  signals 
intelligence  (SIGINT)  and  image  intelligence  (IMINT); 

•  Formal  intelligence  sharing  groups,  such  as  Interdepartmental  Experts  Groups  (IEG);  and 

•  Personal  informal  networks  of  peers  (consisting  of  other  analysts,  experts  on  the  topic,  etc.) 
that  an  analyst  develops  over  time. 

Some  managers  stressed  the  significance  of  the  informal  networks  in  information  sharing 
among  analysts. 

There  are  formal  channels  for  sharing  information,  for  example,  IT  AC  ( the 
Integrated  Threat  Assessment  Centre)  but  in  reality,  it’s  much  more  informal 
networks  of  people  who  respect  each  other,  who  think  they  are  getting  value 
added.  (Interviewee) 

Discussion.  The  nature  and  quality  of  information  available  to  analysts  play  a  significant  role  in 
determining  the  accuracy  of  the  resulting  intelligence  assessments  (Bruce  2008,  Pritchard  and 
Goodman  2009).  In  their  search  for  relevant  data,  analysts  rely  on  the  available  open  and 
classified  information  systems  which  contain  large  quantities  of  information,  much  of 
which  is  of  uncertain  quality.  Commenting  on  the  variety  of  intelligence  information  sources 
(e.g.,  F1UMINT,  SIGINT,  IMINT,  etc.),  Fledley  (2007)  observed  that  “complicating  the  mix  of 
these  sources  of  intelligence  reporting  is  its  sheer  volume,  its  rapid-fire  receipt,  the  ever-present 
‘noise’  of  contradictory  and  inaccurate  information,  and  deliberate  deception  designed  to 
mislead”  (p.213).  In  addition,  the  information  that  is  available  through  different  sources  and  on 
which  analysts  rely  is  not  exactly  “raw”  data,  but  information  that  has  already  undergone  a  certain 
degree  of  processing  and  interpretation.  For  example,  image  data  is  captured  and  interpreted 
before  it  can  be  used  by  an  all-source  analyst;  human  intelligence  is  gathered,  evaluated,  and 
reported  by  intelligence  collectors,  who  themselves  often  receive  information  second,  third,  or 
even  fourth  hand  (Butler  et  al.  2004,  Pritchard  and  Goodman  2009,  Woods  et  al.  2002).  Every 
level  of  processing  of  information  introduces  an  additional  layer  of  interpretation  and  potential 
bias  due  to  limitations  of  collection  instruments,  available  tools,  and  human  interpretation;  “as  a 
result  the  information  that  feeds  into  the  subsequent  analysis  is  never  an  exact  representation  of 
reality”  (Marrin  and  Clemente  2005,  p.714).  Information  that  analysts  use  in  their  assessments  is 
a  product  of  a  whole  chain  of  reporting.  Thus,  analysts  may  not  have  direct  access  to  certain 
individuals  to  get  additional  information  when  needed  (Butler  et  al.  2004,  Fledley  2007,  Pritchard 
and  Goodman  2009).  Analysts’  informal  relationships  with  peers,  which  take  time  to  develop, 
may  allow  them  to  track  information  down  the  reporting  chain  more  effectively  than  may  be 
possible  through  formal  channels. 

In  addition  to  the  information  available  through  various  systems,  there  is  a  great  deal  of  social 
interaction  involved  in  intelligence  analysis  (Johnston  2005).  As  some  interviewees  have  pointed 
out,  even  though  formal  channels  exist,  analysts  rely  mostly  on  their  informal  networks  to 
evaluate  and  interpret  incoming  information,  negotiate  meaning,  and  to  share  alternative  views. 

Despite  the  advantages  of  informal  networks  to  provide  timely  support  to  intelligence  analysts, 
nevertheless,  they  pose  difficulty  from  an  organizational  perspective.  The  drawback  of  informal 
networks  is  that  they  are  individual-dependent  and  not  role-  or  position-dependent,  and  therefore, 
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are  not  recorded  in  organizational  memory  or  supported  by  organizational  procedures.  As  a  result, 
if  an  analyst  moves  to  a  new  position,  the  person  who  replaces  him  or  her  may  not  be  able  to 
establish  a  similar  network.  The  effectiveness  of  informal  networks  depends  on  the  person’s 
willingness  to  work  collaboratively,  which  may  be  determined  by  individual  agendas  and  the 
personality  characteristics  of  the  participants.  Thus,  if  analysts  rely  on  informal  networks,  there  is 
little  an  organization  can  do  to  document  and  influence  these  networks. 

There  has  been  an  effort  in  the  US  IC  to  promote  the  development  of  working  relationships 
among  analysts  by  creating  The  Analyst  Resource  Catalogue  that  was  compiled  by  The  Office  of 
the  Director  of  National  Intelligence  (ODNI).  This  catalogue  contains  some  17,000  analysts  and 
allows  users  to  locate  individuals  working  on  specific  areas  (Tucker  2008).  This  initiative  may 
make  access  to  other  analysts  easier  (depending  on  how  well  it  is  updated  and  maintained),  but  it 
does  not  guarantee  that  a  productive  and  lasting  relationship  will  be  established.  Such  a  directory 
may  be  a  useful  way  of  monitoring  capability  distribution  in  the  community  and  may  allow  faster 
access  to  experts.  It  may  also  be  beneficial  for  the  Canadian  IC  to  explore  the  utility  of 
undertaking  a  similar  initiative. 

3.1 .6.2  Evaluation  of  information 

Results.  Most  managers  pointed  out  that  an  important  step  in  the  intelligence  production 
process  is  the  evaluation  of  the  information  upon  which  analysts  rely  to  make  their  assessments 
and  judgments. 

After  analysts  gather  all  that  information  from  different  sources  what  they  have 
to  do  is  take  all  of  that  information,  open  and  classified  at  different  levels,  and 
they  have  to  bring  it  all  together  and  distill  it.  Part  of  that  distillation  process 
would  be  balancing  and  weighing  evidence  as  it  comes  in.  It  greatly  depends  on 
the  individual  analyst’s  skill  set,  guidance  and  supervision.  It’s  not  done  in  a 
very  formal  way.  (Interviewee) 

Usually,  analysts  working  on  a  given  subject  area  have  formed  a  certain  understanding  of  the 
situation  and  events.  New  information  that  they  gather  for  their  assessments  might  support  their 
theory,  challenge  it,  or  show  divergent  trends.  How  an  analyst  evaluates  new  information  will 
determine  whether  or  not  it  will  be  incorporated  into  his  or  her  analysis  and,  therefore,  might 
affect  the  resulting  judgments.  That  is,  information  might  be  discarded  if  it  is  judged  to  be 
unreliable  or  the  source  is  judged  not  to  be  credible.  Information  collected  through  various 
specialized  channels  (e.g.,  HUMINT,  IMINT,  SIGINT)  is  supposed  to  be  evaluated  for  reliability 
and  credibility  by  information  collectors  or  initial  information  processors.  Analysts  take  these 
evaluations  into  account  when  they  assess  the  information  themselves.  However,  one  manager 
pointed  out  that  intelligence  collectors  do  not  always  evaluate  information  sources,  and  thus 
accuracy  of  the  information  is  unclear.  Some  managers  noted  that  because  there  is  no  standard 
way  to  evaluate  information,  the  onus  is  on  the  analyst  to  assess  its  quality  and  to  decide  whether 
to  include  or  exclude  it  from  consideration.  The  processes  that  analysts  follow  in  assessing 
information  quality  depend,  in  part,  on  their  experience  and  skill  set. 

Discussion.  Quality  and  availability  of  information  are  significant  factors  in  producing  reliable 
intelligence.  As  the  quantity  of  information  available  to  analysts  increases,  so  too  does  the 
amount  of  ‘noise’  (i.e.,  irrelevant  or  unreliable  information)  and  contradiction.  In  addition  to 
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having  to  sift  through  information  of  poor  quality,  analysts  need  to  be  vigilant  to  the  possibility  of 
deception  (Marrin  and  Clemente  2005).  That  is,  they  need  to  carefully  consider  the  relevance  and 
accuracy  of  information.  Based  on  a  recognized  standard  in  the  community,  each  piece  of 
information  is  given  a  rating  on  two  dimensions: 

•  credibility  of  the  source  and 

•  reliability  of  the  information  (e.g.,  Krizan  1999,  ODNI’s  Intelligence  Community  Directive 
#  203  issued  in  2007). 

Clandestine  information  is  assessed  on  these  dimensions  by  the  information  collectors  or 
processors  and  analysts  take  this  information  into  account  in  their  own  evaluation  of  the 
information.  When  an  intelligence  assessment  is  based  on  a  source  that  is  unverified  or  of  low 
credibility,  an  intelligence  failure  may  result,  such  as  in  the  assessment  on  WMD  in  Iraq  (Jervis 
2009,  Schum  and  Morris  2007).  Processes  and  tools  have  been  proposed  to  improve  source  and 
information  evaluation  methods.  For  example,  Schum  and  Morris  (2007),  drawing  on  the 
experience  from  the  legal  system  in  evaluating  the  credibility  of  witness  testimony,  described  the 
computer-assisted  system,  MACE  (Method  for  Assessing  the  Credibility  of  Evidence)  that 
provides  a  systematic  way  of  evaluating  the  source  credibility  and  the  reliability  of  information 
through  answering  a  series  of  questions. 

However,  as  discussed  in  the  previous  sub-section,  analysts  use  information  that  is  a  product  of  a 
(sometimes  lengthy)  reporting  chain.  Information  reliability  and  source  credibility  ratings  are 
(usually)  assigned  by  intelligence  collectors  and  initial  processors  of  that  information;  analysts  do 
not  always  have  access  to  the  information  sources  and  individuals  assigning  these  ratings.  In  fact, 
according  to  Hulnick  (2006),  the  lack  of  communication  between  analysts  and  collection  officers 
is  one  of  the  major  problems  in  intelligence  production.  Moreover,  the  ratings  provided  to 
analysts  are  not  always  accurate  because  they  are  not  always  assigned  in  a  timely  manner  or,  as 
one  interviewee  noted,  if  a  given  source  is  used  repeatedly,  the  source’s  rating  may  not  be 
updated  on  a  regular  basis.  Thus,  intelligence  analysts  must  sometimes  rely  on  their 
technical  knowledge  of  the  collection  processes  or  of  a  particular  reporting  chain  to 
determine  the  credibility  of  a  source.  Informal  networks  can  be  helpful  in  accessing  nodes  in  the 
reporting  chain. 

It  is  worth  noting  that  information  evaluation  and  analysis  are  highly  interdependent,  especially 
when  the  evaluation  is  carried  out  by  analysts.  Besides  assessing  the  credibility  and  reliability  of 
information,  analysts  also  employ  their  understanding  of  the  situation  and  the  presence  or  absence 
of  corroborating  evidence  to  judge  the  value  of  information.  Their  evaluations  determine  whether 
or  not  information  will  be  considered  and  if  it  will  affect  the  assessments.  Although  not  discussed 
in  the  interviews,  it  would  be  interesting  to  determine  in  future  interviews  whether  or  not  a 
feedback  loop  exists  between  analysts  and  collectors.  That  is,  analysts’  evaluations  and  use  of 
information  (which  may  be  based  on  different  criteria  than  collectors’)  could  be  provided  back  to 
collectors  to  allow  them  to  compare,  evaluate,  and,  perhaps,  adjust  their  own  assessments  of  their 
information  sources  and  collection  strategies  accordingly. 
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3.1 .6.3  The  role  of  classification  level  in  information  selection 


Results.  Due  to  the  limited  duration  of  our  interviews  and  the  variety  of  topics  covered,  we  did 
not  discuss  in  detail  how  analysts  search  for  information  and  what  strategies  they  use.  However, 
some  managers  commented  that  information  search  (among  other  factors)  is  affected  by  attributes 
of  the  information  itself,  such  as  its  classification  level.  These  managers  raised  two  main  issues 
with  respect  to  information  classification.  First,  there  are  no  universal  guidelines  across 
organizations  for  assigning  a  classification  level  to  information,  therefore,  different  organizations 
may  use  different  guidelines.  The  resulting  inconsistencies  in  classification  practices  may 
potentially  over-restrict  analysts’  access  to  useful  information.  Second,  some  managers 
noted  that  analysts  tend  to  be  over-reliant  on  secret  (vs.  open)  information  in  their 
assessments,  perhaps  because  classified  information,  being  more  exclusive,  attracts  the  attention 
of  intelligence  consumers. 

Analysts  might  well  generate  a  similar  product  from  open-source  data  only,  but 
reference  to  secret  information  appears  to  draw  attention  to,  and  increase 
credibility  of,  the  reports.  (Interviewee) 

Discussion.  Our  interviewees  pointed  out  that  classified  information  attracts  more  attention  and 
consideration  from  analysts  and  consumers.  However,  level  of  classification  may  or  may  not 
correspond  to  the  value  and  quality  of  information.  For  instance,  a  piece  of  information  from  a 
HUMINT  source  may  be  classified  and,  at  the  same  time,  be  unreliable,  especially  if  the 
intelligence  collector  has  not  properly  assessed  its  credibility  and  reliability. 

The  tendency  to  evaluate  the  importance  of  evidence  based  on  its  classification  level  is  not  unique 
to  the  Canadian  IC,  as  it  was  also  observed  in  the  US  (Johnston  2005,  Lieberthal  2009,  Treverton 
2001).  Classified  information,  due  to  its  exclusivity,  may  create  an  impression  of  importance.  It  is 
also  believed  to  have  more  face  validity  than  information  available  through  open  sources 
(Johnston  2005).  The  root  causes  of  this  phenomenon  are  unclear.  For  example,  Treverton  (2001) 
suggested  that  overreliance  on  secret  information  within  the  IC  may  be  a  heritage  of  the 
intelligence  practices  during  the  Cold  War  era.  He  suggests  that  during  that  period,  most  of  the 
useful  information  came  via  special  and  classified  channels  due  to  the  closed  nature  of  the 
adversary.  However,  we  suspect  that  the  tendency  to  overinflate  the  value  of  classified 
information,  which  we  have  termed  the  “secrecy  bias”  (see  McLellan  et  al.  2008)  ,  reflects  more 
general  features  of  human  information  processing  and  judgment.  That  is,  much  like 
representativeness  and  availability  may  be  used  heuristically  as  proxies  of  subjective  probability 
(for  overviews,  see  Gilovich  et  al.  2002,  Kahneman  et  al.  1982)  so  might  the  secrecy  of 
information  be  used  as  a  proxy  for  its  importance  or  probative  value.  Scientific  research  might 
help  to  provide  a  better  understanding  of  the  nature  of  this  putative  phenomenon  and  ways  to 
counter  its  effects  should  it  be  shown  to  be  a  basis  for  judgments  of  evidential  importance  in 
intelligence  analysis. 

3.1.7  Analysis 

Results.  Selected  bits  and  pieces  of  information  are  used  by  analysts  to  answer  pertinent 
intelligence  questions  of  their  assessments.  Managers  from  both  organizations  commented  that 
the  process  of  intelligence  analysis  is  not  guided  by  formal  procedures  but  rather  relies  mostly  on 
the  individual  analyst’s  approach  and  preference. 
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...  essentially  the  approach  of  analysis  across  the  community  is  the  intuitive 
approach:  you  read  a  lot,  sit  down,  and  write.  (Interviewee) 

Processes  of  analysis  are  part  custom  and  part  rules.  There  are  no  standard 
procedures  for  conducting  intelligence  analysis.  (Interviewee) 

According  to  some  of  the  managers,  there  is  no  one  common,  agreed-upon  methodology  that  can 
be  formalized  and  taught  to  analysts.  Through  a  number  of  training  sessions,  analysts  are  exposed 
to  various  analytic  tools  and  techniques  aimed  at  systematising  the  analysis  process.  However,  the 
application  of  these  methods  requires  additional  time  investment,  is  not  often  re-enforced  by 
organizational  procedures,  and  is  largely  at  the  discretion  of  analysts.  In  addition,  several  CDI 
managers  commented  that  not  much  attention  is  devoted  to  the  analytic  process  in  their 
organization  because  priority  is  placed  on  the  outcomes  rather  than  on  the  process. 

There  is  no  formed  process  that  analysts  follow  in  terms  of  methodology,  and 
each  analyst  approaches  it  slightly  differently.  The  focus  is  on  conclusions,  and 
not  a  lot  of  time  is  spent  on  thinking  about  the  analytical  process.  (Interviewee) 

Some  managers  also  expressed  their  concern  over  the  quality  of  the  analytical  process. 

My  feeling  is  that  too  often  in  the  intelligence  community,  judgments  are  made 
too  often  without  rigorous  thought.  (Interviewee) 


Discussion.  Along  with  information  search  and  evaluation,  analysis  of  the  information  is  one  of 
the  key  activities  in  the  intelligence  process  that  analysts  carry  out.  Analysis  is  the  process  of 
evaluating  and  interpreting  evidence,  piecing  together  (sometimes  seemingly  unrelated) 
information,  generating  explanations  and  alternative  hypotheses,  critically  examining  those 
hypotheses,  and  making  judgments.  Intelligence  analysis  is  predominantly  a  mental  activity 
(Heuer  1999)  and  is  the  key  process  in  arriving  at  intelligence  judgments.  Indeed,  “it  is  by 
thinking  that  analysts  transform  information  into  intelligence”  (Moore  and  Krizan  2003,  p.  1 1 3). 
In  the  context  of  intelligence  analysis,  the  analyst’s  thinking  is  largely  an  unobservable  process, 
which  does  not  leave  a  traceable  path,  unless  the  analyst  deliberately  keeps  a  record  of  his  or  her 
train  of  thought.  Most  of  the  time,  only  the  outcome  of  analyst’s  thinking  will  be  evident  in  the 
form  of  judgments  or  arguments  recorded  in  the  final  report.  Therefore,  it  is  not  surprising  that 
efforts  to  evaluate  and  improve  intelligence  tend  to  focus  on  more  tangible  outcomes  and 
processes,  such  as  descriptions  of  the  quality  of  underlying  sources,  proper  expressions  of 
uncertainties,  and  the  use  of  alternative  analysis.  Although  these  measures  are  intended  to  direct 
and,  perhaps  improve  analysts’  thinking,  they  do  not  directly  capture  analysts’  mental  activity  and 
the  train  of  thought  that  goes  into  undertaking  an  intelligence  assessment. 

Although  intelligence  analysis  has  not  received  much  attention  in  the  scientific  and  practitioner 
literature,  it  has  attracted  public  attention  in  recent  years.  Johnston  (2005)  pointed  out  that  “the 
Intelligence  Community,  in  its  culture  and  mythos  and  in  its  literature,  tends  to  focus  on 
intelligence  operations  rather  than  on  intelligence  analysis”  (p.17,  emphasis  original).  Similarly, 
Bruce  and  George  (2008)  noted  a  relative  shortage  of  literature  devoted  to  intelligence  analysis. 
Johnston  shares  the  view  of  Marrin  and  Clemente  (2006)  who  observed  that  “intelligence  analysis 
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has  historically  been  practiced  more  as  a  craft  reliant  on  the  intrinsic  skill  and  expertise  of  the 
individual  analysts  than  as  a  highly  developed  profession”  (p.642).  Johnston  further  suggested 
that  the  lack  of  substantive  attention  to  the  process  of  analysis  is  (partially)  responsible  for  the 
lack  of  formal  analytic  processes  accepted  and  practiced  in  the  community. 

In  the  current  situation,  analysts,  perhaps  with  their  managers,  subjectively  determine  appropriate 
approaches  and  methodologies  on  a  case-by-case  basis  and  strive  for  analytic  rigour  based  on 
their  own  understandings  of  what  constitutes  analytic  rigour.  This  subjective  approach  to 
methodology  application  may  be  the  only  viable  currently  available  alternative  to  deal  effectively 
with  the  variety  and  complexity  of  analytic  problems.  However,  managers’  concerns  regarding 
the  suboptimal  quality  of  analysis  in  part  also  may  be  due  to  this  approach.  That  is,  the  absence  of 
standard  procedures,  diminished  attention  to  the  analytical  processes,  and  pressure  to  produce  the 
final  report  may  lead  to  cognitive  shortcuts  and  leaps  in  reasoning,  which  in  turn  result  in  inferior 
intelligence  products.  Quality  of  analysis  was  also  one  of  the  most  frequently  mentioned  concerns 
by  analysts  and  managers  in  the  study  by  Treverton  and  Gabbard  (2008)  of  the  US  intelligence 
community. 

The  lack  of  formalized  analytic  procedures  is  not  unique  to  the  Canadian  community.  Johnston 
(2005)  observed  a  similar  situation  in  the  US  community,  pointing  out  that  methods  employed  in 
analysis  are  traditionally  referred  to  as  “tradecraft,”  which  implies  that  “the  methods  and 
techniques  of  analysis  are  informal,  idiosyncratic,  unverifiable,  and  perhaps  even  unexplainable” 
(p.  1 8).  Johnston  further  suggested  the  need  for  documenting,  formalizing,  and  validating  analytic 
methods  used  in  the  community.  It  is  worth  noting  that  certain  steps  in  documenting  and 
formalising  various  analytic  methods  have  been  undertaken  -  see,  for  example,  collections  of 
analytic  techniques  recently  published  by  the  Canadian  CDI  (Thompson  2010),  US  Central 
Intelligence  Agency  (CIA  2005,  2009),  and  the  US  Defence  Intelligence  Agency  (DIA  2008). 
However,  efforts  to  validate  analytic  techniques  objectively  have  been  scarce2  and  this  issue 
remains  pertinent  to  the  present  research  project. 

Whether  or  not  it  is  possible  and  beneficial  to  methodically  structure  the  analytic  process  remains 
an  open  question.  However,  the  study  of  current  practices,  and  validation  of  new  and  existing 
analytic  methods,  certainly  has  merit  for  determining  their  utility. 

3.1.8  Communication 

Results.  Most  managers  stressed  the  opinion  that  effective  communication  is  crucial  in 
intelligence  analysis.  Consumers  of  intelligence  assessments  have  multiple  demands  on  their 
attention,  and  they  have  limited  time  to  attend  to  intelligence  reports.  Managers  stressed  that  it  is 
crucial  for  analysts  to  be  able  to  communicate  effectively  and  efficiently.  Because  decision 
makers’  time  is  scarce,  analysts  need  to  be  able  to  extract  key  judgments  from  reports 
and  present  these  clearly.  Some  managers  also  indicated  that  the  ability  to  communicate 
effectively  is  more  important  to  successful  intelligence  production  than  fluency  in  various 
analytic  tools  and  techniques. 


2  Certain  aspects  of  the  Analysis  of  Competing  Hypotheses  (ACH)  technique  have  been  subjected  to 
scientific  scrutiny  (see  Folker  2000,  Cheikes  et  al.  2004,  and  Pirolli  2006).  However,  the  results  of  these 
studies  are  inconclusive. 
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Communication  is  the  core  of  our  business.  An  analyst  has  to  have  those 
communication  skills.  It  doesn’t  matter  how  smart  the  person  is,  if  you  can’t 
communicate  it  will  be  very,  very  difficult.  Intelligence  is  not  done  for  personal 
benefit,  it  is  done  for  someone  else ’s  benefit,  and  an  analyst  has  to  be  able  to 
communicate;  it’s  a  critical  element.  (Interviewee) 

Some  managers  also  commented  on  the  role  of  language  in  analysis.  The  choice  of  words  to 
convey  the  results  of  an  assessment  determines  the  meaning  being  communicated.  For  instance, 
adjectives  and  qualifiers  used  in  vernacular  in  fact  are  statements  of  judgment  in  intelligence 
reports  and  convey  intended  or  unintended  meaning.  Some  managers  stressed  that  analysts  have 
to  be  cautious  in  selecting  appropriate  words  to  express  the  intended  meaning,  and  the  challenge 
for  them  is  to  be  able  to  back  up  all  of  their  judgments  with  sound  logic  and  evidence  while 
avoiding  unfounded  information. 

Reports  are  a  description/narrative  that  provide  context  and  try  to  simplify 
reality  as  much  as  possible.  Subtleties  of  text  such  as  word  usage,  metaphors, 
and  adjectives  are  all  very  important  and  provide  colour  to  a  report...  .  I 
see  intelligence  analysis  as  a  craft  close  to  literature.  Words  matter,  style 
matters.  (Interviewee) 

Some  managers  also  commented  on  the  difficulty  with  communicating  uncertainty  and 
confidence  of  judgments  in  analytic  products. 

There  is  a  huge  problem  of  language  used  to  convey  probability  and 
importance/magnitude  in  terms  of  what  the  expressions  mean  to  different  people. 
(Interviewee) 

Recognizing  the  vagueness  of  language  and  the  difficulty  it  creates  for  communicating 
uncertainty,  one  manager  pointed  out  that  a  division  at  IAS  uses  numerical  probabilities  to 
communicate  judgment  uncertainty;  numerical  probabilities  are  subsequently  mapped  onto  a 
defined  set  of  verbal  probability  equivalents  in  the  final  reports. 

Discussion.  Communication  is  an  integral  part  of  intelligence  analysis,  as  without  communication 
the  results  of  intelligence  assessments  could  not  be  conveyed  to  decision  makers.  The  necessity  to 
communicate  effectively  is  one  of  the  widely  recognized  requirements  for  intelligence  analysts 
(e.g.,  Gardiner  2009,  Hedley  2007,  Moore  and  Krizan  2003,  ODNI  2009).  Communication  skills 
is  one  of  the  six  performance  elements  for  intelligence  professionals  in  ODNI’s  Intelligence 
Community  Performance  Standards  (ODNI  2009).  An  analyst  could  perform  a  state-of-the-art 
assessment  leading  to  very  important  outcomes.  However,  if  the  analyst  cannot  effectively 
communicate  the  results  of  his  or  her  assessments  and  their  significance,  the  work  would  be  in 
vain  at  best,  and  might  result  in  an  intelligence  failure  (Moore  and  Krizan  2003).  Furthermore, 
Gardiner  (2009)  suggests  that  effectiveness  of  communication  also  depends  on  how  the  analyst’s 
reporting  style  fits  consumer  needs.  Gardiner  suggests  that  the  style  needs  to  be  tailored  to  each 
consumer’s  preferences  to  ensure  better  product  reception.  Therefore,  analysts  may  need  to  be 
able  to  vary  their  reporting  style  to  augment  communication  effectiveness. 

Effective  communication  of  uncertainty  that  an  analyst  has  with  respect  to  his  or  her  judgments  is 
crucial  for  conveying  the  intended  meaning  (Kent  1964).  Verbal  terms  to  denote  probability 
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(e.g.,  “likely,”  “small  chance,”  etc.)  are  imprecise  and  open  to  interpretation,  thus  creating  a 
potential  for  miscommunication.  Kent  (1964)  recognized  this  issue  and  since  then  efforts  have 
been  made  to  standardize  terminology  used  to  communicate  uncertainty,  by  associating 
vernacular  terms  with  numerical  probabilities.  For  instance,  National  Intelligence  Estimates 
produced  by  ODNI  include  “An  Explanation  of  Estimative  Language”  section  with  a  chart 
intended  to  clarify  ambiguity  associated  with  probabilistic  terms.  Adopting  standard  terminology 
across  the  community  may  reduce  the  ambiguity  in  communication,  but  coordination  within  the 
IC  does  not  guarantee  that  intelligence  consumers  will  interpret  the  chosen  terms  in  the  same  way 
as  they  were  intended.  Although  some  research  has  been  conducted  that  examines  how 
intelligence  analysts  interpret  verbal  probabilities  (e.g.,  Mandel  and  Wulf  2010,  Wulf  2008),  to 
the  best  of  our  knowledge,  there  is  no  comparable  study  of  interpretations  of  verbal  probabilities 
among  intelligence  consumers.  Such  a  study  would  be  fascinating  to  conduct,  but  is  not  very 
feasible  given  the  inaccessibility  of  high-level  decision  makers  for  research  purposes.  A  first  step 
that  is  feasible  would  be  to  assign  analysts  to  either  consumer  or  producer  roles. 

The  level  of  precision  with  which  uncertainty  is  identified  is  also  a  debated  issue.  For  example, 
Steinberg  (2008)  argues  that  although  knowing  the  relative  degree  of  uncertainty  is  helpful, 
assigning  probabilities  (or  converting  numerical  probabilities  to  designated  terms)  may  be 
misleading  and  give  a  false  sense  of  concreteness  to  a  consumer.  While  we  agree  that  precise 
numeric  estimates  are  unnecessary  in  cases  where  ordinal  rankings  are  all  that  are  needed  to 
support  decision  making  (e.g.,  in  risk  management  prioritization  exercises),  we  believe  that 
objections  to  the  use  of  numeric  estimates  in  intelligence  analysis  based  on  the  “false  sense  of 
precision”  argument  are  also  somewhat  misguided  since  numeric  probability  estimates  need  not 
be  given  in  precise  terms.  Range  estimates  (e.g.,  70%  -  90%  chance  of  X  happening  in  the  next  6 
months)  are  imprecise,  but  are  nevertheless  unambiguous  and  clear.  Indeed,  such  estimates  also 
communicate  an  analyst’s  level  of  confidence.  Namely,  confidence  is  inversely  proportional  to 
the  range. 

Communication  skills  are  required  in  many  professions.  In  intelligence  analysis,  however, 
communication,  especially  written  communication,  is  a  key  activity.  In  addition  to  merely 
transmitting  information,  written  communication  shapes  an  analyst’s  thinking  process:  “Oral 
briefings  are  valued  and  often  called  for.  But  ultimately,  writing  is  what  the  analyst’s  work  is 
about  -  writing  based  on  organizing  material,  conceptualising,  and  thinking  critically  about  it” 
(Hedley  2007,  p.216). 

3.1 .9  Evaluation  of  quality  of  intelligence  products 

Results.  Some  managers  commented  on  the  lack  of  formalized  procedures  for  evaluating  the 
quality  of  analytic  reports.  These  managers  noted  that  it  does  not  mean  that  quality  checks  are  not 
done  at  all  -  managers  do  check  and  challenge  reports  and  the  underlying  analytic  processes; 
however,  the  quality  assessment  of  intelligence  reports  is  a  subjective  process,  and  specific 
requirements  and  expectations  may  vary  among  managers. 

No,  [there  is  no  standard  for  evaluating  quality  of  the  reports]  other  than  the 
director’s  subjective  view  of  whether  that  product  is  a  quality  product  or  not.  But 
that’s  a  very  loose  standard  and  different  directors  will  have  different  views  of 
that.  (Interviewee) 
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In  addition,  some  IAS  managers  noted  that  producing  an  intelligence  assessment  involves  a  great 
deal  of  interaction  and  collaboration  between  the  analyst  and  his  or  her  manager.  Consequently, 
managers  may  become  involved  in  the  analytic  process  itself,  and,  in  this  case,  analytic  products 
become  the  joint  product  of  the  analyst  and  his  or  her  manager.  Some  managers  observed  that 
manager’s  direct  involvement  in  shaping  the  analytic  product  and  process  could  make  it  difficult 
for  the  manager  to  evaluate  the  quality  of  the  report  objectively. 

...  it  [the  intelligence  report]  is  ultimately  a  joint  product;  it ’s  not  just  the  analyst. 

The  director  should  have  a  role  in  refining  it  and  revising  it.  It’s  difficult  for  the 
[manager]  to  be  completely  objective  because  they  are  in  it  as  well.  (Interviewee) 

The  results  of  our  interviews  suggest  that  the  degree  of  managers’  involvement  in  the  analytic 
process  varies  across  intelligence  organizations.  Managers  from  CDI  mentioned  that  they  may  not 
have  as  much  opportunity  to  get  involved  in  the  analytic  process  due  to  short  timelines  and 
various  pressures  that  require  their  attention. 

Managers  from  IAS  pointed  out  that  their  organization  has  recently  implemented  a  practice  of 
referencing  sources  in  reports  as  a  step  towards  ensuring  rigour  and  quality  of  the  products. 
Analysts  are  required  to  clearly  separate  statements  of  fact  from  their  judgments.  Further,  they 
must  reference  their  sources  for  all  statements  of  fact  and  provide  justifications  for  key  judgments 
in  the  report.  According  to  some  managers,  this  practice  will  make  it  easier  to  trace  back  to  the 
information  on  which  the  analysis  is  based. 

CDI  managers  we  interviewed  commented  that  similar  measures  have  not  been  implemented  in 
CDI,  primarily  because  the  workload  and  time  pressure  do  not  allow  for  steps  that  require 
additional  time. 

Analysis  is  not  procedural  at  CDI.  CDI  analysts  and  their  managers  do  not  spend 
much  time  on  formulaic  processes  ( e.  g. ,  no  footnotes  or  references );  conclusions 
are  what  matter.  Reports  are  vetted,  checked,  challenged,  but  not  through  a 
formal  procedure.  (Interviewee) 

Discussion.  Because  intelligence  is  a  service  function  for  decision  makers,  assessing  quality  and 
value  of  intelligence  products  is  an  important  issue  for  the  intelligence  community.  Moore  and 
Krizan  (2003)  suggested  that  “success”  of  intelligence  analysis  may  be  assessed  by  examining 
two  main  criteria  -  the  intelligence  process  and  intelligence  product  (see  also  Moore  et  al.  2005). 
Assessing  the  “intelligence  product”  implies  assessing  the  value  of  analytic  conclusions  in 
meeting  consumer  needs  (i.e.,  enriching  their  understanding  and  knowledge  regarding  an  issue 
pertinent  to  their  area  of  concern),  as  well  as  the  accuracy  and  calibration  of  the  judgments 
offered  (Mandel  2009a,  Rieber  2004).  Evaluating  the  “intelligence  process”  implies  assessing  the 
soundness  of  methods  used  to  arrive  at  analytic  conclusions,  including  the  train  of  logic,  quality 
of  information,  soundness  of  assumptions,  consideration  of  alternatives,  and  clarity  of 
communication.  Tetlock  and  Mellers  (2009)  discussed  implications  of  putting  emphasis  on  either 
one  of  these  criteria  (i.e.,  product  or  process)  and  suggested  that  a  combination  of  the  two  criteria 
may  provide  a  more  comprehensive  approach  to  evaluating  intelligence  outcomes  and  ensuring 
analysts’  accountability. 
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Brei  (2005)  proposed  six  principles  or  core  values  of  intelligence:  accuracy  of  information  used 
in  analysis;  objectivity  in  judgments;  usability  of  intelligence  by  the  consumers;  relevance  of 
intelligence  to  consumer  requirements;  readiness  of  intelligence  to  respond  to  consumer’s  needs; 
and  timeliness  of  intelligence  (i.e.,  that  it  is  still  actionable  when  delivered).  Brei’s 
principles  are  interdependent  and  violation  of  one  of  them  (e.g.,  accuracy)  may  affect  the 
others  (e.g.,  relevance).  Brei  argued  that  while  some  of  the  six  criteria  do  rely  on  feedback  from 
consumers,  other  criteria  could  be  evaluated  without  such  feedback. 

The  practice  of  source  referencing  employed  by  IAS  is  a  first  step  towards  establishing  standard 
procedures  in  the  process  of  analysis.  It  necessitates  that  analysts  check  the  sources  of 
information  upon  which  they  rely,  making  it  easier  for  managers  to  follow  up  on  the  information 
if  needed.  This  requirement  may  improve  the  quality  of  the  analytic  process  by  reducing  the 
amount  of  time  a  manager  needs  to  invest  in  evaluation;  it  may  also  provide  the  manager  with 
some  information  on  the  analysts’  trains  of  logic.  However,  the  process  by  which  the  information 
was  evaluated,  selected,  interpreted  and  combined  still  remains  subjective,  undocumented,  and  at 
the  discretion  of  the  analyst.  Consequently,  the  evaluation  of  the  product  and  process  still  relies 
mostly  on  the  subjective  judgments  of  the  managers. 

Assessing  usability,  relevance,  timeliness,  and  impact  of  an  intelligence  product  relies  heavily  on 
feedback  from  consumers.  Some  managers  commented  that  it  is  difficult  for  them  to  obtain 
information  on  the  impact  their  finished  products  have  on  their  consumers  due  to  the  paucity  of 
feedback.  Measuring  the  impact  of  analytic  products  is  indeed  a  difficult  problem,  and  not  only 
because  of  limited  feedback.  Because  the  role  of  intelligence  is  to  provide  a  better  situational 
understanding  (and  not  to  suggest  decisions)  to  consumers,  the  impact  of  intelligence  on  the 
resulting  consumer’s  decisions  cannot  be  easily  judged  and  measured  (Herman  1996,  Treverton 
2001).  In  addition,  consumers  receive  information  from  various  sources  (intelligence  being  only 
one),  which  makes  it  hard  to  attribute  a  difference  in  someone’s  understanding  to  a  specific 
source  of  information  (Medina  2009,  Treverton  2001). 

Herman  (1996)  and  Bruce  and  George  (2008)  discussed  the  problems  with  measuring  the  real 
impact  of  intelligence  assessments:  It  is  difficult  to  assess  the  “quality”  of  information  in  a  report, 
and  it  is  difficult  to  estimate  the  impact  of  a  given  intelligence  product  on  a  particular  action. 
“Most  intelligence  effects  are  on  users’  frame  of  mind  rather  than  on  identifiable  actions.  Even 
where  particular  intelligence  outputs  can  be  correlated  with  use  there  is  usually  no  obvious  way  of 
measuring  effects  on  outcomes”  (Herman  1996,  p.299).  Also,  measuring  the  effect  of  information 
is  difficult:  “as  for  its  use,  no  one  really  knows  what  difference  information  makes”  (p.300).  The 
difficulty  in  measuring  intelligence  success  also  stems  from  the  fact  that  resulting  changes  in 
policymakers’  views  or  preventative  actions  inevitably  change  the  state  of  affairs  with  which  the 
intelligence  was  concerned  (Betts  2009,  Bruce  and  George  2008).  Thus,  it  makes  it  difficult  to 
envision  the  counterfactual  “what  if’  consequences  of  analysis  on  the  development  of  events. 

Managers  and  analysts  may  make  efforts  to  ensure  the  relevance  of  their  reports  at  the  outset  of 
the  assessments  by  defining  and  negotiating  topics  for  the  reports  and  timelines  for  their  delivery; 
however,  it  does  not  guarantee  that  the  final  products  will  be  timely,  relevant,  and  useful. 

Besides  meeting  consumer  needs  for  information,  another  important  characteristic  of  intelligence 
products  is  the  actual  accuracy  of  the  judgments  provided  in  the  reports,  which  may  be  used  in 
evaluating  analytic  reports.  Judgment  accuracy  is  different  from  Brei’s  principle  of  accuracy, 
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which  refers  to  the  information  used  in  analysis.  The  latter  stresses  the  accuracy  of  inputs  used  in 
analysis,  and  the  former  refers  to  the  accuracy  of  predictions  or  the  output  of  the  analysis  process. 
Although  misjudgements  are  inevitable  in  intelligence  analysis  (Brady  1993,  Heuer  1999), 
intelligence  products  would  have  been  of  little  use  if  their  assessments  were  only  seldom  or 
randomly  accurate. 

As  in  other  areas  where  probability  estimates  are  provided  by  professionals  (Dawson  et  al.  1993, 
Murphy  and  Winkler  1984,  Tetlock  2005),  the  accuracy  of  predictive  intelligence  judgments  can 
be  examined  systematically  over  large  numbers  of  cases  using  objective,  quantitative  measures 
such  as  calibration  and  discrimination  analyses  (Yaniv  et  al.  1991).  An  analyst  is  said  to  be  well 
calibrated  if  the  relative  frequencies  of  observed  events  match  the  assigned  probabilities  of 
predicted  events.  In  other  words,  for  judgments  of,  say,  0%,  20%,  40%,  60%,  80%,  and  100% 
chances  of  occurrence,  perfect  calibration  would  occur  when  the  relative  frequencies  in  these 
probability  classes  were  0%,  20%,  40%,  and  so  on,  respectively.  Discrimination,  on  the  other 
hand,  measures  the  extent  to  which  an  analyst  uses  the  full  range  of  the  probability  scale  and 
avoids  hedging  his  or  her  bets  by  always  predicting  the  base  rate  within  a  judgment  class  (i.e.,  the 
average  likelihood  of  predicted  events  actually  occurring)  or  a  fudge  value  such  as  “fifty-fifty” 
(Fischhoff  and  Bruine  de  Bruin  1999).  Statistical  measures  of  discrimination  are  akin  to  measures 
of  the  proportion  of  variance  explained  in  the  criterion.  Although  any  given  probabilistic 
prediction  is  unfalsifiable,  calibration  and  discrimination  measures  offer  a  means  of  assessing  the 
aggregate  performance  of  an  analyst  or  analytic  organization.  Aggregate  analyses  of  predictive 
accuracy  also  avoid  the  problems  of  cherry  picking  successes  (Tetlock  2009)  and  misconstruing 
particular  intelligence  failures  as  a  general  failure  of  the  system  (Betts  1978).  In  Canada,  Mandel 
(2009b)  conducted  such  an  analysis  of  the  accuracy  of  approximately  600  predictive  judgments 
made  by  analysts  in  one  IAS  division.  He  found  that  analysts’  judgments  exhibited  a  high  degree 
of  calibration  and  good  discrimination.  Indeed,  excluding  a  small  number  of  “fifty-fifty” 
judgments,  about  90%  of  the  predicted  events  were  correctly  classified.  Correct  classification  in 
this  case  means  that  events  don’t  occur  when  the  assigned  probability  of  the  predicted  event  is 
less  than  .5  (or  5/10  on  the  relevant  0-10  scale  used)  and  events  do  occur  when  the  assigned 
probability  is  greater  than  .5.  Stated  differently,  in  about  90%  of  the  cases  examined  the 
probabilistic  prediction  offered  pointed  in  the  right  direction.  Similar  studies  have  yet  to  be 
conducted  in  other  countries  or  with  other  intelligence  organizations  within  Canada.  Thus,  for  the 
time  being,  the  quantitative  analysis  of  judgment  accuracy  (and  quality,  more  generally)  has  been 
very  limited. 

It  is  worth  noting  however,  that  compared  to  other  areas  where  probability  estimates  are  provided 
by  professionals  (e.g.,  the  medical  profession),  intelligence  judgments  have  a  unique  property  - 
an  intelligence  judgment  can  lead  consumers  to  implement  preventative  actions  that  may  alter  the 
likelihood  of  the  predicted  event’s  occurrence  or  prevent  it  all  together  (Betts  2009).  In  light  of 
this  “warning  problem,”  one  might  expect  a  greater  degree  of  deviation  in  analysts’  accuracy 
(Mandel,  2009a). 

Calibration  of  analysts’  accuracy  can  only  assess  analysts’  performance  with  respect  to  the  events 
that  analysts  identified.  However,  it  has  no  means  of  incorporating  occurrence  of  relevant  events 
that  analysts  did  not  identify  or  foresee,  but  which  have  a  significant  impact  on  consumers.  An 
overall  measure  of  intelligence  judgment  quality  also  needs  to  take  into  account  what  proportion 
of  the  set  of  all  relevant  events  was  identified  by  intelligence  judgments.  A  comprehensive 
assessment  of  intelligence  judgment  quality  may  need  to  incorporate  a  set  of  measures  that 
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capture  different  aspects  of  analysis  quality.  For  instance,  Mandel  (2009b)  examined  calibration 
and  discrimination  as  a  function  of  the  importance  of  the  analytic  judgment  for  intelligence 
consumers  (as  judged  by  an  independent  senior  intelligence  analyst).  Interestingly,  judged 
importance  had  a  negligible  effect  on  those  measures  of  judgment  quality,  indicating  that  it  is  not 
simply  the  case  that  performance  is  good  for  relatively  inconsequential  judgments. 

It  is  the  responsibility  of  intelligence  managers  and  analysts  to  ensure  the  soundness  of  the 
analytic  process  and  resulting  judgments.  Evaluating  the  quality  of  intelligence  processes  is 
closely  related  to  the  state  of  the  current  practice  of  intelligence  analysis  in  the  community. 
Intelligence  analysis  has  been  practiced  as  a  craft,  mostly  relying  on  the  skills,  capabilities,  and 
experience  of  individual  analysts.  There  is  a  lack  of  accepted  standard  analytic  procedures  and 
therefore,  evaluation  of  analytic  processes  is  subjective  (Johnston  2005,  Marrin  and  Clemente 
2006).  As  some  managers  pointed  out,  this  subjective  evaluation  allows  for  variability  in  the 
criteria  used  by  different  managers. 

In  a  situation  where  there  are  no  standard  evaluation  procedures  and  it  is  difficult  to  obtain 
feedback  from  consumers,  subjective  appraisal  by  managers  is  often  the  only  product  evaluation 
that  takes  place.  Some  managers  commented  that  the  subjectivity  of  the  product  evaluation  may 
be  furthered  by  managers’  involvement  in  the  development  of  a  product.  Managers’  contributions 
to  shaping  intelligence  products  make  it  more  difficult  for  them  to  remain  objective  when 
evaluating  it.  On  one  hand,  a  manager’s  involvement  in  the  product  development  process  helps  to 
ensure  that  the  product  conforms  to  the  manager’s  quality  requirements  before  the  product  is 
disseminated.  On  the  other  hand,  a  certain  degree  of  personal  authorship  in  the  product  might 
make  it  more  difficult  for  the  manager  to  assume  a  completely  objective  perspective  while 
evaluating  the  product’s  overall  quality  and  impact,  especially  if  other  more  objective  measures 
are  unavailable. 

In  addition,  evaluation  of  analytic  assessments  is  a  time-consuming  process  that  requires 
significant  investment  on  the  part  of  the  manager.  Depending  on  their  time  constraints,  managers 
may  not  have  time  to  adequately  evaluate  each  report.  Demands  on  managers’  time  are 
determined  by  the  number  of  reporting  analysts,  reports’  timelines,  and  production  volume.  When 
these  demands  increase,  a  manager,  naturally,  will  have  less  time  available  for  each  report.  The 
nature  of  activities  undertaken  by  an  organization  in  part  shapes  an  organization’s  structure  and 
control  mechanisms.  Mintzberg  (1979)  identified  five  possible  coordinating  mechanisms  within 
an  organization  that  shape  its  structure:  mutual  adjustment,  direct  supervision,  and  standardization 
of  inputs,  processes,  and  outputs.  The  discussion  above  suggests  that  standardization  in 
intelligence  analysis  at  any  level  -  inputs,  processes,  or  outputs  -  is  not  feasible  and,  perhaps, 
may  not  be  beneficial.  Intelligence  organizations  rely  mainly  on  managers  directly  supervising 
analysts  and  their  activities,  and  on  mutual  adjustment  through  informal  interaction  among 
analysts  working  as  a  team.  Mutual  adjustment  and  direct  supervision  require  significant 
involvement  from  managers  in  their  subordinates’  activities  and  greatly  limit  their  optimal  span  of 
control.  In  order  for  these  coordinating  mechanisms  to  be  exercised  effectively,  one  manager 
ought  to  supervise  a  relatively  small  unit.  As  more  analysts  join  an  organization,  further  division 
may  be  required  to  maintain  coordination  at  an  optimal  level  and  avoid  overloading  managers. 
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3.2  Challenges  in  intelligence  analysis 


The  nature  of  analytic  activities  and  the  organizational  environment  place  certain  demands  on 
analysts.  How  efficiently  one  can  function  under  these  demanding  conditions  depends  on 
individual  experiences  and  one’s  ability  to  cope  in  the  analytic  environment.  This  section 
discusses  a  number  of  individual  and  organizational  challenges  in  the  analytic  process  that 
managers  identified. 

3.2.1  Challenges  stemming  from  the  nature  of  the  task  and  individual 
differences 

3.2.1 .1  Information  overload 

Results.  Most  managers  noted  that  one  of  the  challenges  for  intelligence  analysts  is  that  the 
amount  of  available  information  is  constantly  growing,  and  analysts  must  sort  through,  evaluate, 
and  absorb  what  is  relevant. 

One  challenge  for  analysts  is  information  overload.  A  large  amount  of  data 
needs  to  be  digested  and  only  relevant  information  extracted  from  it. 
(Interviewee) 

The  processes  of  information  search  and  analysis  are  highly  interdependent.  Some  managers 
pointed  out  that  analysts  have  to  walk  the  fine  line  between  collecting  too  little  information  (thus 
making  flawed  inferences  based  only  on  what  is  available)  and  collecting  additional  information 
at  the  expense  of  engaging  in  thorough  analysis. 

The  challenge  is  to  know  when  to  stop  searching  and  to  start  drawing 
conclusions  given  the  existing  uncertainties.  (Interviewee) 

Some  managers  commented  that  there  are  no  guidelines  for  determining  how  much  information  is 
enough,  and  analysts  make  this  decision  intuitively  based  on  their  experience.  The  amount  of  time 
available  to  produce  a  report  constrains  and  determines  the  duration  of  information  search. 

[Question:  How  do  you  decide  that  you  have  enough  information  and  that  you 
can  start  drawing  conclusions?]:  There  is  no  real  way  to  frame  this  one.  I  guess 
it's  instinctive.  I  start  writing  when  I  feel  confident  in  my  ability  to  address  an 
issue.  Yet,  deadlines,  more  than  anything  else,  usually  dictate  the  writing 
schedule.  (Interviewee) 

In  addition,  all  managers  commented  that  intelligence  analysis  is  characterised  by  a  great 
deal  of  uncertainty,  and  that  analysts  need  to  compensate  for  gaps  in  available 
information  with  their  judgments. 

Discussion.  Rapid  development  and  expansion  of  communications  and  information  technologies 
considerably  facilitate  creation,  transmission,  and  storage  of  information  leading  to  the  constant 
increase  of  information  available,  which  impacts  intelligence  analysts  and  the  process  of  analysis 
(Hedley  2007,  Johnson  2007,  Johnston  2005,  Treverton  2001,  Woods  et  al.  2002).  However, 
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individual  ability  to  extract  meaning  from  that  information  has  not  increased  along  with  the 
available  quantity  of  information  (Woods  et  al.  2002).  Quantity  of  data  cannot  compensate  for  its 
quality.  On  the  contrary,  some  managers  commented  that  often  there  is  a  lot  of  information 
available  on  a  given  issue,  but  some  of  it  is  not  credible  and  a  lot  of  missing  pieces  remain.  When 
more  information  is  made  available,  analysts  have  to  sort  through  more  data  to  find  what  is 
relevant  and  reliable. 

“The  challenge  for  intelligence  -  sorting  fact  from  fiction,  or  signals  from  noise  -  is  new  only  in 
magnitude.  But  the  change  in  magnitude  is  awesome  ...  in  some  respects,  the  harder  problem  for 
intelligence  arises  simply  from  the  volume,  not  evil  intent:  As  ‘publishing’  gets  easier,  standards 
of  verification  go  down.  Collecting  information  is  less  of  a  problem,  and  verifying  it  is  more  of 
one”  (Treverton  2001,  p.9  -  p.10).  Analysts’  knowledge  of  the  issue  at  hand  helps  them  to 
effectively  deal  with  the  mass  of  information  (Davis  1992,  Woods  et  al.  2002).  However,  because 
requests  to  intelligence  analysts  have  become  more  varied,  analysts  are  more  often  required  to 
work  on  topics  outside  their  bases  of  expertise.  This  exacerbates  the  problem  of  information 
overload,  as  analysts  do  not  have  sufficient  background  knowledge  to  help  them  separate  signals 
from  noise  (Patterson  et  al.  2001). 

Time  constraints  make  it  virtually  impossible  for  analysts  to  absorb  all  available  information,  and 
there  is  a  trade  off  between  having  ample  information  and  providing  a  timely  assessment.  The 
risk  associated  with  stopping  the  search  too  soon  is  that  important  information  might  be  missed 
and  the  resulting  analysis  and  judgments  misguided.  Prolonging  the  search  for  too  long,  however, 
might  not  allow  sufficient  time  for  processing  and  analyzing  the  information,  or  the  analysis 
might  reach  decision  makers  too  late  to  be  useful,  especially  in  the  case  of  warning  intelligence 
(Betts  1978).  Prolonging  the  information  search  can  also  be  a  symptom  of  decision  avoidance, 
discussed  in  the  next  sub-section.  In  a  study  by  Patterson  et  al.  (2001),  experienced  analysts  who 
were  under  time  pressure  missed  some  relevant  information  and  had  difficulty  resolving  data 
conflicts  when  analysing  an  unfamiliar  topic.  However,  analysts  who,  despite  the  pressure, 
spent  more  time  searching  for  information  and  reading  more  documents  made  fewer  or  no 
inaccurate  statements. 

Medina  (2009)  also  points  out  that  information  “abundance”  has  made  it  more  difficult  for 
intelligence  analysts  to  provide  value  to  intelligence  customers.  Medina  indicates  that  intelligence 
consumers  are  generally  well  informed  and,  in  addition  to  other  sources  of  information  (such  as 
media  and  personal  communications),  they  often  have  access  to  “raw”  incoming  traffic  of 
intelligence  at  the  same  time  as  the  analysts.  Having  access  to  “raw”  intelligence  traffic  before 
they  receive  analysts’  assessments  of  this  information,  intelligence  consumers  inevitably  interpret 
and  form  their  own  understanding  of  the  available  facts  before  analysts  have  a  chance  to  provide 
their  insights.  Thus,  to  provide  value,  analysts  have  to  “surpass  the  analytic  abilities  of  their 
customers”  (Medina  2009,  p.  1 10)  and  also,  perhaps  change  any  judgments  they  may  have  made  if 
they  do  not  concur. 

With  more  information  available,  analysts  may  have  less  confidence  that  they  have  sampled  all 
relevant  corroborating  and  contradictory  information  that  is  potentially  available,  and  they  do  not 
have  time  to  exhaust  their  sources  (Woods  et  al.  2002).  Information  overload  also  taxes  analysts’ 
cognitive  resources,  as  they  need  to  search,  evaluate  and  absorb  increasing  volumes  of 
information.  Information  overload  places  higher  demands  on  analysts’  memory  and 
representational  abilities,  which  are  needed  in  order  to  discern  or  “visualize”  patterns  in  vast 
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quantities  of  information.  Learning  effective  search  and  information  management  and 
organization  strategies  becomes  important  to  this  end.  We  did  not  discuss  in  our  interviews  with 
managers  the  efficiency  of  analysts’  information  search  strategies,  but  there  are  some  indications 
in  the  literature  that  analysts  may  be  unaware  of  the  range  of  available  information  search 
techniques.  For  example,  Patterson  et  al.  (2001)  observed  that  analysts,  whose  analytic  experience 
ranged  between  seven  and  thirty  years,  mostly  used  inefficient  “primitive”  search  strategies. 
Patterson  et  al.  suggested  that  focussed  training  and  better  design  of  information  systems  may 
increase  the  effectiveness  of  analysts’  search  behaviours.  Some  managers  commented  that  there  is 
a  lack  of  adequate  information  systems  that  help  to  organize  and  connect  relevant  information. 
According  to  Woods  et  al.  (2002)  “  ...  solving  data  overload  problems  requires  both  new 
technology  and  an  understanding  of  how  systems  of  people  supported  by  various  artefacts  extract 
meaning  from  data”  (p.34). 

With  respect  to  continuous  striving  for  more  information  in  intelligence  analysis,  Heuer  (1999) 
raises  an  issue  of  added  value  of  acquiring  additional  information,  recommending  thorough 
evaluation  of  the  available  evidence  instead  of  continuous  search  for  additional  data.  Heuer 
suggests  that  intelligence  analysis  is  analogous  to  medical  diagnosis  in  that  an  analyst  should 
generate  several  plausible  explanations  given  the  available  evidence  and  then  collect  additional 
information  that  will  be  diagnostic  in  differentiating  between  the  hypotheses  in  the  considered  set, 
much  as  a  physician  will  attempt  to  do  in  reaching  a  medical  diagnosis.  Marrin  and  Clemente 
(2005)  support  Heuer’ s  medical  diagnosis  analogy,  and  suggest  that  the  analytic  profession  might 
stand  to  benefit  from  adopting  current  practices  used  by  the  medical  profession  to  maximize  the 
accuracy  of  diagnoses. 

3.2.1 .2  Judgment  avoidance  -  fear  of  being  wrong 

Results.  Most  managers  commented  that  analysts  are  required  to  make  judgments  in  all  of  their 
reports,  which  is  a  difficult  task  for  analysts. 

Some  analysts  can’t  make  a  judgment  partly  because  they  don’t  want  to  be 
wrong.  (Interviewee) 

There  is  a  lot  of  uncertainty  about  the  conclusion,  and  it  is  often  hard  to  make  a 
decision.  (Interviewee) 

Some  managers  attributed  the  difficulty  in  making  judgments  mainly  to  the  uncertainty  inherent 
in  analytic  activity  and  the  fear  of  being  wrong.  According  to  these  managers,  decision  avoidance 
is  common  among  intelligence  analysts  because  their  decisions  often  have  implications  for  human 
lives.  Further,  variations  in  individual  personality  characteristics  may  contribute  to  difficulty  in 
making  judgments. 

I’ve  seen  this  very  clearly.  Complete  avoidance  of  judgments  or  caveating  (sic)  a 
judgment  to  the  point  that  no  matter  what  happens  they  will  be  right. 
(Interviewee) 

Discussion.  Judgment  avoidance  may  be  exhibited  in  either  withholding  judgment  altogether,  that 
is,  not  providing  judgment  on  an  issue  at  all,  or  it  may  also  be  expressed  by  providing  a  judgment 
but  in  a  vague  form  that  is  consistent  with  several  interpretations.  As  some  managers  pointed  out, 
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making  judgments  in  intelligence  assessments  is  difficult  because  situations  and  their  outcomes 
are  uncertain,  and  there  is  always  a  possibility  for  a  judgment  to  turn  out  to  be  mistaken. 
Judgment  avoidance  may  be  due  to  the  desire  to  escape  accountability  for  a  judgment  that  may  be 
incorrect  or  an  effort  to  achieve  consensus  with  other  analysts  or  agencies,  which  tends  to  make 
the  judgments  even  more  vague. 

Intelligence  is  a  service  that  is  intended  to  provide  useful  information  to  intelligence  consumers. 
Some  practitioners  suggested  that  vague  judgments  are  not  helpful  to  consumers,  and  instead  of 
striving  for  consistency  among  analysts  and  agencies,  it  may  be  more  beneficial  to  highlight 
alternative  views  or  potential  outcomes  (McLaughlin  2008,  Steinberg  2008). 

3.2.1 .3  Reluctance  to  accept  alternative  perspectives 

Results.  Another  difficult  aspect  of  intelligence  analysis  according  to  some  managers  is  for 
analysts  to  have  their  judgments  challenged  by  their  managers  or  colleagues. 

It  is  difficult  [for  analysts]  to  accept  other  ideas  and  thoughts  and  allow  people 

to  challenge  their  conclusions.  (Interviewee) 

Nevertheless,  managers  have  to  challenge  their  analysts  as  part  of  their  role  to  ensure  quality 
control  of  the  products.  For  managers,  challenging  their  analysts  requires  maintaining  a  balance 
between  questioning  their  conclusions  to  ensure  rigour  and  quality  in  the  analytic  products  and 
trusting  in  their  analysts’  expertise. 

Discussion.  Analysts’  resistance  to  having  their  assessments  challenged  could  be  explained  by  a 
number  of  different  psychological  causes,  some  of  which  are  personality-driven  (e.g.,  high  Need 
for  Closure;  Webster  and  Kruglanski  1994)  and  some  that  are  common  to  the  process  of  challenge 
itself.  At  the  time  the  analyst’s  work  is  challenged,  the  analyst  has  presumably  invested 
considerable  time  and  effort  into  the  work  to  settle  on  certain  conclusions.  At  this  point,  it  might 
be  difficult  to  consider  alternatives  because  most  people  have  an  inherent  desire  to  maintain 
intrapersonal  consistency  (Cialdini  et  al.  1995).  Analysts  might  be  inclined  to  resist  challenges  to 
their  conclusions  because  inconsistency  (i.e.,  changing  one’s  conclusions)  feels  unnatural  or 
uncomfortable.  There  are  conceivably  a  number  of  self-enhancement  biases  that  could  also  cause 
analysts  to  resist  the  challenge  function.  Generally  speaking,  self-enhancing  beliefs  can  be  quite 
adaptive.  For  example,  being  confident  in  one’s  ability  can  breed  persistence,  which  can  in  turn 
lead  to  success  (Taylor  1989).  Positive  illusions  (i.e.,  positive  self-views  that  may  be  unfounded) 
are  also  associated  with  benefits  such  as  adaptive  coping  and  improved  intellectual  functioning 
(Taylor  and  Brown  1988).  Taylor  and  Brown  identified  three  types  of  positive  illusions  that 
people  tend  to  exhibit:  unrealistically  positive  views  of  their  abilities,  exaggerated  control  over 
events,  and  excessive  optimism  about  their  future.  The  first  type  especially  could  stifle  the 
challenge  process.  When  a  manager  challenges  the  analyst’s  work,  an  analyst  may  feel  that  the 
manager  is  questioning  his  or  her  judgment  and  reasoning  abilities.  It  may  be  difficult  for  analysts 
to  consider  that  they  might  be  wrong  when  they  hold  such  positive  illusions  about  their  analytical 
abilities.  Further,  people  tend  to  be  overconfident  in  their  factual  judgments  -  especially  for 
difficult  problems  -  which  can  lead  them  to  dismiss  opposing  views  (Fischhoff  1991).  The 
challenge  process  is  difficult  not  only  for  the  analysts  but  also  for  their  managers.  Managers  often 
do  not  have  as  much  background  information  on  the  issue  or  as  much  time  to  devote  to 
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it  as  analysts  do,  and  so  it  is  difficult  to  generate  pertinent  questions  that  will  truly  serve  the 
challenge  function. 

3.2.1 .4  Poor  logic  and  a  lack  of  rigour  in  analysis 

Results.  Some  managers  expressed  their  concern  over  the  quality  of  analytic  thinking  that  goes 
into  some  intelligence  assessments  and  the  overall  analytic  rigour. 

Sometimes  analysts  draw  false  conclusions  whereby  the  evidence  is  precise  but 
the  inference  unfounded.  Basic,  informal  logic  is  very  important.  (Interviewee) 

The  rigour  in  challenging  hypotheses,  and  search  for  information  that  disproves 
hypotheses  is  not  there  ...  they  [analysts]  are  given  that  much  [a  lot  of] 
information  to  read  and  are  told  to  produce.  They  don ’t  have  time  to  go  through 
a  rigour.  (Interviewee) 

Discussion.  No  doubt,  the  quality  of  logic  and  analytic  rigour  are  constrained  by  the  individual 
abilities  of  an  analyst  and  his  or  her  knowledge  and  experience.  Some  of  these  individual 
characteristics  include  analytic  reasoning  skills  and  knowledge  of  state-of-the-art  analytic 
techniques  and  their  appropriate  application.  Some  authors  suggested  that  analysts  require 
creative  imagination  to  envision  alternative  hypotheses  and  explanations  (Bruce  2008,  Pritchard 
and  Goodman  2009).  The  degree  of  rigour  that  an  analyst  employs  depends  on  his  or  her 
understanding  of  what  constitutes  good  analytic  practice,  which  may  vary  considerably  due  to  the 
community’s  reliance  on  an  “intuitive  approach”  to  analysis  and  the  lack  of  accepted  analytic 
standards.  In  addition,  as  our  interviewees  pointed  out,  overwhelming  situational  demands  (i.e., 
short  timelines  coupled  with  massive  amounts  of  information)  may  preclude  even  the  best 
equipped  analyst  from  ensuring  analytic  rigour  and  following  best  practices.  As  a  result,  the  lack 
of  analytic  rigour  and  poor  logic  in  analysis  may  be  the  product  of  various  causes  such  as, 
individual  characteristics,  accepted  analytic  standards  within  the  community,  and  situational 
demands.  Addressing  this  problem,  therefore,  requires  analyzing  and  identifying  the  relative 
contribution  of  these  causes  to  the  problem  of  the  lack  of  rigour  in  a  particular  case,  and  it  will 
most  likely  require  a  variety  of  intervention  strategies. 

3.2.1 .5  Cognitive  biases 

Results.  Most  managers  pointed  out  that  analysts  are  susceptible  to  cognitive  biases,  which  affect 
thinking,  logic,  and  ultimately  the  resulting  judgments,  and  that  overcoming  these  cognitive 
obstacles  is  not  easy. 

Intelligence  analysis  is  a  business  of  forecasting,  predicting  the  future,  and  the 
analysts  tend  to  focus  on  the  status  quo  -  assume  the  future  will  look  like  the 
present  ....  Analysts  are  prisoners  of  their  own  experiences.  There  are  problems 
of  confirmation  bias  and  mirror  imaging,  and  it  is  very  hard  to  avoid  them. 
(Interviewee) 
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There  is  a  whole  aspect  of  trying  to  understand  a  different  culture  and  [to]  think 
from  their  perspective  on  what  is  likely  to  happen  next.  That  is  a  huge  challenge, 
because  it  is  completely  and  utterly  foreign.  (Interviewee) 

One  manager  also  commented  on  the  negative  relationship  between  the  level  of  an  analyst’s 
expertise  and  his  or  her  ability  to  recognise  the  changing  trends  in  a  situation  (i.e.,  to  evade 
confirmation  bias). 

Some  longer-term  analysts  have  developed  a  way  of  looking  at  things  and  may 
not  give  sufficient  weight  to  new  things  that  challenge  the  way  they  have  been 
dealing  with  an  issue  in  the  past.  Sometimes,  newer  analysts  can  make  better 
distinctions  about  things  that  have  changed.  (Interviewee) 


Discussion.  The  role  of  cognitive  biases  and  nrind-sets  in  intelligence  analysis,  and  especially  in 
intelligence  failures,  is  one  of  the  key  topics  that  has  attracted  the  community’s  attention.  It  has 
been  generally  accepted  that  nrind-sets  and  biases  are  unavoidable,  as  they  are  inherent  to  human 
information  processing  (Butterfield  1993,  Davis  1992,  Heuer  1999).  Davis  further  noted  that: 

“  ...  no  amount  of  forewarning  about  the  confirmation  bias  (belief  preservation),  the 
rationality  bias  (mirror  imaging),  and  other  powerful  but  perilous  shortcuts  for  processing 
inconclusive  evidence  that  flow  from  the  hardwiring  of  the  brain  can  prevent  even  veteran 
analysts  from  succumbing  to  analytic  errors.  One  observer  linked  cognitive  biases  to 
optical  illusions;  even  when  an  image  is  so  labelled,  the  observer  still  sees  the  illusion” 
(p.159). 

In  the  literature,  cognitive  biases  have  also  been  divided  into  those  that  are  inherent  to  human 
information  processing,  such  as  confirmation  bias,  and  those  that  arise  from  one’s  knowledge  and 
experience  with  an  issue,  such  as  one’s  mental  model  (mind-set)  or  worldview  (Davis  2008). 
Both  types  of  bias  direct  and  affect  an  analyst’s  information  processing  and  are  seen  as 
unavoidable.  In  addition,  according  to  Davis,  the  mindsets  that  are  developed  over  the  course  of 
one’s  experience  with  a  topic  are  “indispensable,”  as  without  them,  the  task  of  analysis  would  be 
impossible:  “an  open  mind  is  as  dysfunctional  as  an  empty  mind”  (Davis  2008,  p.  160). 

The  difficulty  associated  with  understanding  a  different  culture’s  perspective  and  projecting  one’s 
own  cultural  values  and  beliefs  onto  others  has  been  referred  to  as  mirror-imaging,  “everybody- 
thinks-like-us  mindset,”  coherence  bias,  rationality  bias,  and  projection  bias  (Davis  2008,  Heuer 
1999).  Intelligence  analysts  often  have  to  understand  and  predict  behaviour  and  motives  of 
individuals  and  groups  from  countries  or  regions  that  have  different  backgrounds,  lifestyles, 
cultures,  values,  and  goals  than  those  of  Western  society  where  analysts  reside.  Making  accurate 
predictions  and  decisions  about  a  foreign  group  or  culture  requires  understanding  values  and 
motivations  of  that  culture  or  group,  and  understanding  the  situation  from  that  culture’s 
perspective.  Some  managers  commented  that,  to  gain  this  understanding,  analysts  tend  to  rely  on 
knowledge  collected  during  area  trips  to  the  region  and  their  direct  experiences  and  immersion  in 
the  culture.  Nevertheless,  it  is  difficult  not  to  project  the  basis  for  one’s  own  reasoning  and 
motivations  onto  the  other  party.  Attribution  of  the  other  party’s  reasoning  and  motivation  based 
on  one’s  own  perspective  (whether  correct  or  erroneous)  may  become  a  form  of  mind-set  for 
understanding  the  opponent.  Misjudgements  based  on  such  erroneous  attributions  may  be  seen  in 
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many  intelligence  failures  such  as  the  failure  to  recognise  Soviet  motivation  and  willingness  to 
install  nuclear  missiles  in  Cuba  preceding  the  Cuban  Missile  Crisis  in  1962,  the  failure  to 
anticipate  the  surprise  attack  that  started  the  Yom  Kippur  War  in  1973,  and  the  failure  of 
the  Iraqi  government  to  foresee  the  US  intervention  in  the  Iraq-Kuwait  conflict  leading  to  the 
1991  Gulf  War  (Brady  1993,  Davis  2008,  Jervis  2009).  Furthermore,  it  may  become  even  more 
difficult  for  analysts  to  overcome  cultural  barriers  if  a  certain  mindset  or  perspective  is 
institutionally  reinforced. 

One  of  the  approaches  to  dealing  with  the  issue  of  mind-sets  and  biases  proposed  in  the  literature 
has  been  focused  on  raising  analysts’  and  managers’  awareness  of  potential  biases  (Heuer  1999). 
Although  increased  awareness  may  not  prevent  seeing  an  illusion  as  an  illusion  (using  Davis’ 
analogy),  it  may  still  increase  awareness  of  the  possibility  of  such  an  illusion  and  lead  to  a  more 
careful  examination  of  one’s  conclusions.  Another  approach  to  help  alleviate  biases  that  has  been 
discussed  in  the  literature  focuses  on  the  development  and  application  of  tools  and  techniques 
(Davis  2008,  Heuer  1999).  An  example  of  a  technique  developed  to  counter  confirmation  bias  is 
ACH  developed  by  Heuer  (1999).  Although  there  is  evidence  to  suggest  that  ACH  does  reduce 
confirmation  bias  in  novice  analysts,  its  effects  on  expert  analysts  has  not  been  demonstrated 
using  scientific  research  methods  (Cheikes  et  al.  2004).  This  further  reinforces  the  necessity  of 
subjecting  developed  tools  and  techniques  to  careful  and  systematic  evaluation  in  order  to  assess 
their  effectiveness  in  countering  certain  cognitive  biases. 

Managers’  observations  of  the  negative  relationship  between  an  analyst’s  expertise  on  an  issue 
and  his  or  her  ability  to  recognise  “unlikely”  developments  was  also  discussed  by  Davis  (2008). 
Davis  suggested  that  analysts’  expertise  in  a  given  domain  may  be  very  helpful  in  foreseeing  the 
normal  course  of  events  but  may  make  it  more  difficult  to  recognise  “unlikely”  developments. 
The  question  is  what  are  the  underlying  psychological  processes  of  the  analyst,  and  the  task- 
specific  properties  of  intelligence  analysis,  that  lead  to  this  effect.  The  difference  between  an 
expert  and  novice  analyst  is  that  the  expert  undoubtedly  has  formed  a  highly  developed 
understanding  of  the  situation  -  a  “mind-set,”  using  Davis’  terminology  -  which  is  probably  more 
intricate  than  that  of  the  novice.  The  expert  is  “deeper  into  the  woods,”  so  to  speak.  In  addition, 
the  “accuracy”  of  the  expert’s  model  has  been  frequently  confirmed  (i.e.,  by  events  that  have 
conformed  to  the  model)  by  virtue  of  the  fact  that  they  have  been  working  on  the  issue  for  a 
longer  period  of  time  than  the  novice.  Naturally,  the  expert  examines  (consciously  or  not) 
incoming  information  in  light  of  the  presently  entertained  theory;  the  novice,  who  may  still  be 
developing  his  or  her  model,  may  have  a  more  “neutral”  approach  to  incoming  information. 

Observations  from  intelligence  experts  raise  an  interesting  issue  regarding  the  relationship 
between  expertise  and  susceptibility  to  the  confirmation  bias,  which  has  not  been  studied 
extensively  in  the  field  of  cognitive  psychology.  There  are  conflicting  views  in  the  scientific 
literature  with  respect  to  the  impact  of  expertise  on  performance  in  novel  situations,  and  the 
underlying  cognitive  processes  involved.  For  instance,  in  the  domain  of  expertise  research,  it  is 
generally  theorised  that  acquired  expertise  in  a  domain  leads  to  greater  sensitivity  to  details  in 
that  specific  domain  (Gobet  and  Simon  1996)  and  increases  cognitive  flexibility,  which  is 
expected  to  attenuate  confirmation  biases  (Feltovitch  et  al.  1984,  Krems  and  Zierer  1994,  Smith 
and  Kida  1991).  Conversely,  research  in  the  domain  of  creativity  and  skill  acquisition  has 
suggested  that  knowledge  of  certain  methods  of  approaching  a  given  problem  and  prior 
experience  with  the  domain  may  actually  lead  to  the  opposite  effect  and  increase  rigidity  in 
experts  that  hinders  them  in  finding  creative  or  novel  solutions.  This  cognitive  rigidity  is 
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presupposed  to  be  the  result  of  set  (Einstellung)  effect  or  automatization  created  through 
experience  (Bilalic  et  al.  2008,  Luchins  1942,  Sternberg  1996).  The  increased  cognitive  rigidity 
does  not  directly  imply  the  presence  of  confirmation  bias,  yet  such  a  relationship  is  plausible. 

It  is  worth  noting,  however,  that  while  there  are  several,  competing  theoretical  views  on  the 
subject  of  cognitive  flexibility  or  rigidity  of  expertise,  the  empirical  evidence  supporting  or 
refuting  these  views  is  rather  scarce  and  inconclusive  (Bilalic  et  al.  2008).  In  addition,  the  domain 
of  intelligence  analysis  may  possess  unique  properties,  not  transferable  to  other  domains. 
Directed  examination  of  the  nature  of  expertise  in  the  intelligence  analysis  field  is  a  promising 
starting  point  in  gaining  a  better  understanding  of  its  impact  on  experts’  interpretation  and 
judgments  of  dynamic  environments. 

The  analytic  challenges  discussed  above  most  likely  do  not  have  ready-made  solutions.  However, 
further  investigation  and  scientific  research  may  improve  our  understanding  of  the  underlying 
social  and  psychological  processes  involved. 

3.2.2  Challenges  due  to  the  organizational  environment 

This  sub-section  outlines  characteristics  of  the  organizational  environment  that  contribute  to 
challenges  in  intelligence  production. 

3.2.2. 1  Inadequate  staffing 

Results.  One  of  the  most  commonly  mentioned  challenges  on  an  organizational  level  by  CDI 
managers  was  not  having  a  sufficient  number  of  personnel  to  perform  all  of  the  activities  in  the 
organization’s  mandate.  This  leads  to  analysts  in  CDI  performing  a  wide  variety  of  tasks. 

Our  people  have  to  work  on  different  things  depending  on  demand,  because  we 
are  short  on  people.  (Interviewee) 

The  issue  of  an  inadequate  number  of  human  resources  was  not  raised  as  much  by  the  IAS 
managers  that  we  interviewed,  and  so  seems  to  be  more  pertinent  to  the  military  environment. 

Discussion.  The  effects  of  having  fewer  personnel  than  is  optimal  for  performing  the  required 
tasks  are  not  unequivocal  and  may  entail  positive  as  well  as  negative  consequences  for  both  the 
organization  and  its  staff.  According  to  staffing  sufficiency  theory  (Barker  1960,  1968,  Vecchio 
and  Sussmann  1981,  Wicker  1979a),  moderate  levels  of  understaffing  (i.e.,  where  there  is 
(moderately)  insufficient  or  barely  sufficient  personnel  to  carry  out  the  essential  tasks)  have  been 
associated  with  increased  employee  motivation  and  involvement,  increased  task  diversity,  skill 
utilization,  increased  individual  effort,  and,  in  some  instances,  improved  individual  and  group 
productivity  (Ganster  and  Dwyer  1995,  Perkins  1982,  Vecchio  and  Sussmann  1981).  Thus,  under 
certain  conditions,  understaffing  may  have  positive  individual  and  organizational  effects.  The 
relationship  between  the  level  of  understaffing  and  task  perceptions  is  curvilinear,  meaning 
that  extreme  levels  of  understaffing  will  result  in  perceived  work  overload  and  stress, 
which  undermines  any  positive  effect  of  understaffing  (Vecchio  and  Sussmann  1981).  In 
addition,  positive  effects  of  moderate  understaffing  may  diminish  with  continuous  exposure  to 
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increased  workload  situations  and  result  in  negative  consequences  (Wicker  and  Kirmeyer  1976, 
Wicker  1979b). 

The  inadequate  staffing  issue  in  CDI  seems  to  be  more  of  a  chronic  nature  and,  theoretically,  may 
lead  to  the  other  problems  listed  below: 

•  Time  pressure  and  work  overload  for  current  personnel,  because  they  have  to  complete 
multiple  tasks.  Time  pressure  not  only  compromises  the  reasoning  process  itself  but,  due  to 
time  constraints,  also  limits  the  tools  and  techniques  that  analysts  are  able  to  utilize  in  their 
analysis  (as  application  of  each  tool  or  technique  requires  additional  time). 

•  Analysts  may  be  unable  to  develop  deep  expertise  in  a  given  area  or  topic,  as  they  are 
constantly  switched  to  address  currently  pressing  issues.  This  may  also  result  in  a  lack  of 
continuity  as  different  individuals  might  work  on  a  given  topic  at  different  times  depending 
on  their  availability. 

•  The  emphasis  shifts  toward  addressing  current  issues,  and  strategic  organizational 
development  concerns  are  more  likely  to  fall  by  the  wayside;  formal  organizational 
processes  are  less  likely  to  be  followed  and  workarounds  are  more  likely  to  appear. 

•  Constant  work  overload  eventually  results  in  employee  stress,  decreased  motivation,  and 
contributes  to  personnel  turnover. 

It  may  be  beneficial  to  explore  optimal  staffing  arrangements  that  may  minimize  negative  impact 
on  employees.  Addressing  this  issue,  however,  requires  wider  organizational  analysis  and 
intervention,  the  feasibility  of  which  depends  on  available  resources,  other  constraints,  and 
existing  competing  priorities. 

3. 2.2. 2  Turnover 

Results.  Another  frequently  mentioned  organizational  challenge  was  the  issue  of  turnover.  This 
issue  was  especially  pertinent  to  CDI  because  a  considerable  proportion  of  CDI’s  analysts  are 
regular  military  personnel  that  are  on  a  three-year  posting  cycle.  One  interviewee  mentioned  that 
about  one  third  of  analytic  staff  at  CDI  change  every  year  due  to  the  regular  military  posting 
cycle.  At  IAS,  turnover  is  less  of  an  issue  because  analysts  are  civilian  indeterminate  public 
servants  that  are  not  affected  by  posting  cycles.  IAS  has  been  able  to  retain  their  analysts  for 
longer  periods,  on  average. 

When  an  analyst  walks  out  the  door  -  assets  are  portable  -  everything  is  between 
the  ears.  You  walk  out  and  it’s  gone.  It’s  really  a  bad  business  model ...  you  need 
a  system  that  will  groom  and  facilitate  long  term  periods  of  analytic  study  on 
areas  without  having  to  deal  with  this  back  and  forth  of  people  coming  and 
going.  The  loss  is  really  too  high;  it’s  too  much  of  an  investment  and  you  get 
nothing  out  of  it  when  they  walk  out  the  door.  (Interviewee) 

Discussion.  Similar  to  understaffing,  turnover  has  been  shown  to  have  complex  impact  on  both 
individuals  and  organizational  performance.  Negative  impacts  of  turnover  include: 

•  general  reduction  in  productivity  (Argote  et  al.  1995,  Bluedorn  1982); 
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•  loss  of  expertise  in  a  subject  area,  organizational  practices,  and  loss  of  organizational 
memory  (Carley  1992); 

•  disruption  of  organizational  processes  (Price  1977,  1989,  Staw  1980); 

•  disruption  of  social  relationships  within  an  organization  that  may  be  imperative  for  task 
performance  (Dess  and  Jason  2001,  Leana  and  Buren  1999);  and 

•  demoralization  among  remaining  employees  who  are  required  to  take  on  additional 
responsibilities  to  cover  for  personnel  who  have  left  the  organization  (Mowday  et  al.  1982). 

On  the  other  hand,  turnover  has  been  credited  with  increasing  organizational  innovation  by 
bringing  in  new  individuals  (Dalton  and  Todor  1979,  Price  1989)  and  improving  the  individual- 
organization  fit.  The  latter  is  mainly  applicable  to  volunteer  turnover  (when  individuals 
choose  to  leave  or  join  an  organization  out  of  their  free  will)  and  it  may  not  be  applicable  to  the 
military  context. 

Given  that  constant  turnover  is  inevitable  and  predictable  in  the  military  environment,  it  may  be 
possible  to  plan  for  and  implement  various  measures  to  minimize  its  negative  impacts. 
Minimizing  the  impact  of  turnover  on  organizational  practices  requires  systematic  knowledge 
preservation  practices  and  ensuring  effective  knowledge  transfer  from  departing  analysts  to  new 
arrivals.  However,  from  our  interviews,  there  seemed  to  be  no  systematic  practices  in  place  at  the 
organizations  surveyed.  Although  there  may  be  several  causes  for  this  situation,  the  issue  of 
inadequate  personnel  plays  a  significant  role.  Implementation  of  systematic  knowledge 
preservation  and  transfer  activities  would  be  beyond  immediate  operational  needs,  and  thus  would 
put  additional  strain  on  already  limited  organizational  resources.  Additional  resources  (i.e., 
organizational  slack)  may  be  necessary  to  implement  any  new  initiatives  while  maintaining  the 
same  level  of  output. 

3.2.2.3  Time  pressure 

Results.  Time  pressure  was  also  identified  as  an  integral  property  of  the  military 
intelligence  environment. 

There  is  constant  time  pressure.  (Interviewee) 

The  issue  of  time  pressure  and  how  that  affects  judgment  and  decision  making  is 
very  relevant.  (Interviewee) 

Time  pressure  creates  a  situation  in  which  various  tasks  that  analysts  carry  out  as  part  of  their 
assessments  are  forced  into  competition.  As  a  result,  there  is  less  time  to  apply  analytic 
techniques,  to  document  the  analytical  process,  and  to  reference  sources.  The  issue  of  time 
pressure  was  raised  more  by  managers  from  CDI  than  managers  from  IAS. 

Discussion.  Conducting  intelligence  analysis  under  constant  time  pressure  may  have  an  impact  on 
the  process  of  analysis  and  its  outcome  as  well.  Time  pressure  affects  information  search  and 
evaluation  practices  as  well  as  analysts’  ability  to  solicit  feedback  from  other  experts  in  the  field. 
Limited  time  to  complete  an  assessment  restricts  the  amount  of  time  available  for  thorough 
analysis  and  evaluation  of  alternative  hypotheses.  Time  pressure  forces  analysts  to  think  quickly, 
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not  allowing  them  to  pay  as  much  attention  to  the  thinking  process  itself  (thus  making  analysts 
potentially  more  susceptible  to  inherent  pitfalls  of  human  judgment). 

The  effects  of  time  pressure  on  the  intelligence  assessment  process  is  two-fold:  On  the  one  hand, 
it  has  a  direct  effect  on  the  analysis  process  by  limiting  the  time  available  for  information  search, 
analysis  and  report  preparation;  and,  on  the  other  hand,  it  affects  thinking  and  decision  making 
processes  (Maule  and  Edland  1997).  There  is  an  opinion  within  the  intelligence 
community  that  time  pressure  may  contribute  to  and  exacerbate  effects  of  cognitive  biases  (Bruce 
and  Bennett  2008,  Johnston  2005,  Treverton  2008).  Research  in  cognitive  psychology  has  shown 
that  time  pressure: 

•  affects  performance  on  tasks  with  high  working  memory  demand  (Beilock  and  Carr  2005); 

•  alters  risk-taking  behaviour  and  reduces  attention  to  secondary  information  (Dror  and 
Busemeyer  1999,  Huber  and  Kunz  2007); 

•  reduces  the  amount  of  information  considered  in  decision  making  (Huber  and  Kunz  2007, 
Rieskamp  and  Hoffrage  2008); 

•  affects  decision  making  and  problem  solving  strategy  (Rieskamp  and  Hoffrage  2008,  Zakay 
and  Wooler  1984);  and 

•  reduces  confidence  in  the  decision  (Smith  et  al.  1982). 

A  research  program  designed  specifically  to  investigate  the  effects  of  time  pressure  on  both  basic 
cognitive  processes  involved  in  intelligence  analysis  and  impact  on  expert  performance  would 
help  to  clarify  the  role  of  time  pressure  on  the  intelligence  analysis  processes. 

3. 2.2.4  Lack  of  feedback  on  the  final  product 

Results.  Managers  from  IAS  commented  that  their  analysts  cannot  always  receive  feedback  on 
the  impact  of  their  assessments  on  the  decision  making  process. 

Getting  feedback  on  the  final  product  is  one  of  the  most  difficult  problems  of  this 
whole  thing  -  to  know  whether  we  are  fully  meeting  needs  of  our  clients. 
(Interviewee) 

This  issue  was  mostly  raised  by  the  IAS  managers,  as  there  seems  to  be  more  of  a  disconnection 
between  analysts  and  decision  makers  than  at  CDI. 

Discussion.  The  main  purpose  of  intelligence  assessments  is  to  provide  critical  information  to 
decision  makers  -  policy  makers  or  commanding  officers  -  to  aid  their  understanding  of  a 
situation  and  allow  them  to  make  more  informed  decisions  (Herman  1996,  Treverton  2001).  The 
relationship  between  the  producers  and  consumers  of  intelligence  assessments  is  critical  in 
ensuring  the  utility  of  these  assessments.  Ideally,  the  input  from  intelligence  consumers  into  the 
intelligence  production  process  is  required  at  minimum  at  two  stages:  during  the  formulation  of 
intelligence  requirements  and  providing  feedback  after  the  assessment  is  completed.  The  role  of 
clear,  specific,  and  timely  requirements  is  important  for  directing  analysts’  efforts  onto  the  right 
path.  Similarly,  intelligence  consumers’  feedback  on  the  utility  of  the  intelligence  product  after  it 
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is  completed  is  essential  for  analysts  to  better  understand  consumers’  needs  and  to  adjust  their 
(future)  assessments  accordingly. 

As  we  discussed  in  Sub-section  3.1.3,  an  intelligence  production  unit  or  agency,  as  any  other 
social  organization,  can  be  conceptualized  as  a  system  with  inputs  (e.g.,  information,  feedback), 
transformation  (e.g.,  the  process  of  analysis),  and  outputs  (e.g.,  completed  intelligence 
assessments).  In  systems  theory,  feedback,  especially  negative  feedback,  is  one  of  the  essential 
informational  inputs  that  allows  a  system  to  adjust  its  processes  in  order  to  successfully  function 
in  its  environment  (Katz  and  Kahn  1978).  Lack  of  adequate  feedback  to  analysts  on  their  products 
threatens  the  relevance  of  the  intelligence  production  system. 

The  relationship  between  producers  and  consumers  of  intelligence  is  complex  and  based  on 
mutual  interdependency.  However,  the  initial  trend  in  the  US  (and,  perhaps,  Canadian) 
community  to  separate  intelligence  from  decision  makers  in  an  effort  to  preserve  analytic 
integrity  and  avoid  “politicization”  of  intelligence  has  threatened  the  relevance  of  intelligence 
products  to  consumers.  This  disconnection  is  reinforced  by  the  structural  organization  of 
intelligence  consumer  and  producer  roles  that  does  not  allow  for  frequent  interaction  and 
collaboration.  In  light  of  recent  reviews  of  intelligence  practice,  many  writers  on  intelligence 
recognize  the  necessity  of  “bridging  the  gap”  between  intelligence  producers  and  consumers  in  an 
effort  to  make  intelligence  more  relevant  (Gardiner  2009,  Steinberg  2008,  Treverton  2001,  2008). 

Another  issue  worth  noting  in  the  producer-consumer  relationship  that  may  contribute  to  the 
difficulty  in  obtaining  feedback  on  final  products  is  that  the  interdependency  between  intelligence 
consumers  and  producers  is  not  symmetrical  (Davis  2006).  Given  that  intelligence  is  only  one  of 
the  sources  of  information  for  consumers  (Davis  2006,  Medina  2009),  intelligence,  being  a 
support  function,  is  more  dependent  on  consumers  than  consumers  are  dependent  on  intelligence. 
This  asymmetric  interdependence,  no  doubt,  contributes  to  communication  challenges 
experienced  between  producers  and  consumers.  Bringing  intelligence  producers  and  consumers 
closer  together  may  require  certain  organizational  restructuring  and  task  adjustments.  This 
relationship  is  extremely  important  for  ensuring  relevance  of  intelligence  products,  and  warrants 
more  focused  attention  in  the  Canadian  community.  For  an  elaborate  analysis  of  consumer- 
producer  relationship  in  the  Canadian  context,  see  Cox  (2010). 

3. 2.2.5  Breakdowns  in  inter-departmental  and  inter-organizational  information 

sharing 

Results.  Some  managers  pointed  out  that  there  are  a  lot  of  breakdowns  in  interdepartmental  and 
inter-organizational  information  sharing,  which  prevent  analysts  from  having  access  to  potentially 
valuable  information. 

I  think  if  we  actually  did  a  forensic  scrub  of  how  we  move  information,  and  who 
we  post  it  to  and  share  it  with,  I’m  sure  there  are  a  lot  of  breakdowns  in  the 
organization  and  that  we  don ’t  pass  key  things,  they  don ’t  pass  key  things.  Each 
pillar  or  subgroup  has  its  own  practices  for  how  they  share  and  that  is  something 
I  wouldn  ’t  know  anything  about.  (Interviewee) 

Nevertheless,  analysts  compensate,  to  some  extent,  for  breakdowns  in  information  flow  by 
maintaining  their  informal  personal  networks  of  peers,  to  whom  they  can  turn  whenever  needed. 
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A  great  deal  of  intelligence  sharing  is  still  a  personal  relationship.  Even  if  it’s 
not  the  right  way  to  go,  they  can  get  you  in  touch  with  the  right  person  because 
they  know  their  organization.  (Interviewee) 

Discussion.  Information  is  by  far  the  most  important  input  in  the  intelligence  analysis  process. 
Analysts  draw  on  information  from  several  sources,  some  of  which  rely  on  sharing  within  their 
own  organization  and  sharing  among  different  organizations.  Herman  (1996)  argued  that 
production  of  effective  intelligence  requires  inter-agency  cooperation.  The  main  barriers  to  free 
information  flow  that  some  managers  commented  on  could  be  attributed  to  existing  inconsistent 
information  classification  practices  in  different  organizations,  the  “need  to  know”  attitude,  which 
results  in  reluctance  to  share  information,  and  breakdowns  in  interdepartmental  coordination  and 
integration.  The  US  intelligence  community  is  faced  with  similar  problems  (Treverton  and 
Gabbard  2008)  and,  perhaps,  even  to  a  greater  extent  than  the  Canadian  community  due  to  the 
vast  number  of  players  in  the  US  arena.  Breakdowns  and  barriers  in  information  sharing  increase 
uncertainty  for  intelligence  analysts  and  can  contribute  to  intelligence  failures.  For  instance, 
Treverton  (2001)  attributed  the  accidental  bombing  of  China’s  embassy  in  Belgrade  in  1999  to 
intelligence  failure  resulting  from  breakdowns  in  communication  between  imagery  analysts  and 
local  collectors.  Also,  breakdowns  in  inter-organizational  information  sharing  and  coordination 
were  identified  as  one  of  the  main  causes  of  the  9/1 1  attack  intelligence  failure  in  the  US 
community  (9/11  Commission  2004,  Hulnick  2008). 

Some  managers  pointed  out  that  information  sharing  in  the  intelligence  community  relies  to  a 
large  extent  on  informal  social  networks.  The  recently  released  Intelligence  Community 
Performance  Standards  document  in  the  US  recognizes  the  importance  of  informal  networks  -  the 
ability  to  develop  and  sustain  such  networks  as  one  of  the  six  performance  elements  for 
professional  intelligence  analysts  (ODNI  2009). 

As  already  discussed  in  Sub-section  3.1.6.1,  “Information  sources”,  there  are  advantages  and 
disadvantages  associated  with  relying  on  informal  networks  for  information  sharing.  Informal 
networks  greatly  depend  on  personal  relationships,  and  it  may  take  a  long  time  for  a  new  analyst 
to  establish  a  network  of  peers.  When  these  relationships  are  disrupted  for  various  reasons  (for 
example,  as  a  result  of  changing  positions),  information  sharing  is  disrupted  as  well. 

The  US  community  has  launched  an  effort  under  the  ODNI  initiative  to  improve  inter- 
organizational  coordination  and  collaboration.  Under  this  initiative,  the  Analyst  Resource  Catalog 
has  been  created,  which  includes  names  and  areas  of  expertise  of  over  17,000  analysts  from 
across  the  community  allowing  specialists  to  find  one  another.  In  addition,  a  central  depository 
for  all  intelligence  assessments  from  across  the  community,  the  National  Intelligence  Library,  has 
been  created  with  the  goal  of  fostering  information  sharing  across  agencies  (Tucker  2008).  Given 
that  these  practices  prove  to  be  beneficial  to  the  US  community,  the  implementation  of  similar 
approaches  might  be  considered  within  the  Canadian  community  as  well. 

It  emerged  from  our  interviews  that  an  extensive  study  of  information  needs,  flows,  and 
breakdowns  within  Canadian  IC  may  help  to  improve  information  flow  in  some  organizations,  as 
well  as  within  the  community  more  broadly. 
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3.3  Essential  skills 


Results.  All  managers  interviewed  agreed  that  successful  performance  of  analytic  tasks  requires  a 
certain  degree  of  inherent  individual  ability.  Our  interviewees  pointed  out  that  identifying  a  set  of 
essential  skills  and  capabilities  required  for  intelligence  analysis  is  an  important  issue  for  the 
community.  Managers  shared  their  ideas  regarding  what  these  essential  skills  may  be.  We 
grouped  the  identified  characteristics  into  four  general  categories  -  analytic  ability,  environmental 
suitability,  knowledge,  and  personal  characteristics.  Table  1  presents  an  aggregated  list,  which 
includes  a  grouped  summary  of  all  skills  that  were  mentioned.  Some  of  the  skills  in  this  list  were 
mentioned  by  only  one  manager  while  other  skills  were  mentioned  by  several.  Most  of  the  items 
are  listed  in  their  original  wording  and  are  arranged  in  alphabetical  order  within  the  categories. 


Table  1:  List  of  essential  skills  for  intelligence  analysts  identified  by  managers 


Skill/Ability  Description 


1  ANALYTIC  SKILLS 


Abductive 
1 . 1  reasoning 
skills 


Abductive  reasoning  refers  to  inferring  the  best  possible  explanation  for 
the  given  evidence.  Abductive  reasoning,  according  to  some  managers, 
creates  new  knowledge,  whereas  inductive  and  deductive  reasoning  do 
not. 


Ability  to  deal 
with  large 
amounts  of 
information 


Refers  to  the  ability  to  extract  and  summarise  information  from  a  large 
quantity  of  available  data.  This  ability  is  perceived  by  managers  to  be  an 
inherent  talent,  which  is  hard  to  teach. 


Analysts  predominantly  have  to  deal  with  incomplete  information,  which 
Awareness  of  requires  making  assumptions  with  respect  to  what  is  missing  in  order  to 
1.3  one’s  produce  judgments.  A  good  analyst  is  aware  of  the  assumptions  upon 

assumptions  which  his  or  her  judgments  rely,  which  allows  for  the  assessment  of  the 
quality  of  his  or  her  assumptions. 


Although  managers  did  not  elaborate  on  what  they  meant  by  critical 
Critical  thinking,  this  term  generally  refers  to  a  process  of  purposeful  thinking 

thinking  about  a  subject  that  employs  logic,  examination  of  the  evidence  and 

assumptions,  and  evaluation  of  the  thinking  process  itself. 
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Writing  and 

1.5  communication 
skills 

No  matter  how  important  the  intelligence  assessment  conclusions  might 
be,  they  will  be  of  limited  value  if  the  analyst  is  unable  to  communicate 
them  in  an  effective  manner.  Because  the  majority  of  intelligence 
assessments  are  communicated  in  written  reports,  analysts  need  to  be 
good  writers. 

2  ENVIRONMENTAL  SUITABILITY 


Ability  to  deal 
2.1  with 

uncertainty 

Uncertainty  is  an  inherent  characteristic  of  intelligence  analysis  and 
analysts  need  to  be  able  to  work  under  conditions  of  chronic  uncertainty 
and  effectively  deal  with  its  consequences.  This  includes: 

•  the  ability  to  make  up  for  missing  information  with  judgments  and 
conscious  assumptions; 

•  to  have  the  drive  to  get  to  the  bottom  of  the  problem  despite  various 
obstacles; 

•  to  overcome  the  fear  of  being  wrong  and  the  discomfort  with 
uncertainty  of  the  decision  outcome  and  make  judgments 

Ability  to  work 

2 2  under 

significant 

stress  levels 

Stress  here  refers  to  emotional  and  mental  stress.  Factors  that  contribute  to 
stress  for  intelligence  analysts  include  time  pressure,  number  of  tasks, 
data  overload,  consequences  of  assessments,  deception,  etc. 

9  „  Be  emotionally 
solid 

This  ability  was  mentioned  in  relation  to  military  analysts  who  deal  with 
operations.  Because  the  consequences  of  intelligence  reports  lead  to  the 
“application  of  violence  on  the  battlefield,”  (and  thus  have  tremendous 
impact  on  other  people’s  lives),  analysts  need  to  be  able  to  emotionally 
manage  the  consequences  of  their  actions  and  remain  at  the  top  of  their 
performance. 

2.4  Teamwork 

Analysts  need  to  be  able  to  work  effectively  in  a  team  environment. 

2  5  Time 

sensitivity 

Intelligence  is  time  sensitive  and  analysts  need  to  be  able  to  accept 
deadlines  and  deliver  final  products  on  time. 

3  KNOWLEDGE 
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^  j  Breadth  of 
experience 

Knowledge  of  various  areas.  This  quality  is  more  pertinent  to  CDI 
analysts,  who  are  moved  around  the  organization. 

^  2  Deep  regional 
expertise 

This  quality  is  more  pertinent  to  IAS,  where  analysts  are  required  to 
specialize  in  a  given  region.  Deep  knowledge  and  firsthand  experience 
with  that  region  is  extremely  beneficial  for  interpreting  events  as  well  as 
incoming  information  from  the  region. 

^  ^  Environmental 
awareness 

Analysts  need  to  have  background  knowledge  in  their  area  and  on  the 
issues  with  which  they  are  dealing;  they  need  to  know  what  is  important 
and  what  to  pay  attention  to  in  the  vast  amount  of  incoming  information. 

Knowledge  of 
3.4  the  collection 
process 

Analysts  need  to  be  knowledgeable  about  various  collection  methods  and 
their  strengths  and  shortcomings.  This  knowledge  impacts  analysts’ 
abilities  to  properly  evaluate  the  reliability  and  credibility  of  information, 
and  is  especially  pertinent  to  information  collected  through  HUMINT 
sources. 

4  CHARACTERISTICS 


4. 1  Accurate 
memory 

Analysts  have  to  read  through  large  quantities  of  documents  and  reports, 
and  keep  the  details  of  what  they  have  read  in  their  memory  in  order  to  be 
able  to  find  patterns  and  connections  among  various  documents.  Analysts 
need  to  rely  on  their  memory  especially  given  the  lack  of  efficient 
information  management  systems. 

4.2  Flexibility 

Flexibility  here  refers  to  the  ability  to  adapt  to  changing  environmental 
(organizational)  demands. 

^  ^  Integrity  and 
moral  courage 

Analysts  need  to  have  enough  courage  to  “speak  truth  to  power”.  This 
may  be  particularly  challenging  in  the  military  context. 

4.4  Self-starter 

Analysts  need  to  be  self-motivated  and  constantly  working  on  their  area. 

^  .  Thirst  for 
knowledge 

Analysts  need  to  be  inquisitive,  and  have  a  passion  for  constant  learning 
and  reading. 

4.6  Thoroughness 

This  refers  to  the  need  to  be  detail-oriented  and  careful,  especially  in 
evaluating  and  interpreting  evidence. 
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Discussion.  Identifying  core  competencies  required  for  intelligence  analysis  is  an  important  issue 
for  the  community  and  is  essential  for  selection  and  performance  evaluation  of  intelligence 
analysts.  Both  the  US  and  Canadian  intelligence  communities  have  started  examining  this  issue  in 
more  detail.  For  example,  DRDC’s  Director  General  Military  Personnel  Research  and  Analysis 
(DGMPRA)  conducted  an  Intelligence  Officer  job  analysis  and  identified  five  competencies 
required  by  Intelligence  Officers  -  general  cognitive  ability,  personality  factors,  leadership, 
information  technology,  and  communication  skills  (Smith  2009).  In  a  follow  on  effort,  Girard 
(2010)  found  a  significant  overlap  between  core  competencies  of  Intelligence  Officer  and 
Intelligence  Operator  occupations.  The  competencies  identified  by  managers  in  our  interviews 
considerably  overlap  with  those  identified  by  Smith  (2009)  and  Girard  (2010).  For  example, 
“general  cognitive  ability”  identified  by  Smith  (2009)  overlaps  with  “analytic  skills”  and 
“knowledge”  categories  in  Table  1.  “Personality  factors”  from  the  study  by  Smith  (2009)  overlaps 
with  “environmental  suitability”  and  “characteristics”  categories  in  Table  1. 

The  US  intelligence  community  has  undertaken  an  effort  to  standardize  performance 
requirements  for  intelligence  analysts.  ODNI  has  recently  released  a  document  outlining  a  set  of 
uniform  performance  standards  for  the  entire  US  intelligence  community  (ODNI  2009).  Specified 
performance  dimensions  overlap  somewhat  with  those  listed  by  managers  in  our  interviews,  and 
include:  accountability  for  results,  communication,  critical  thinking,  engagement  and 
collaboration,  personal  leadership  and  integrity,  and  technical  expertise.  In  order  to  be  applicable 
to  the  entire  intelligence  community,  the  list  is  inevitably  general,  but  it  captures  the  essential 
competencies  required  for  intelligence  analysis. 

The  list  compiled  from  managers’  comments,  naturally,  is  not  a  comprehensive  list  of  analysts’ 
skills;  however,  it  provides  an  insight  into  the  skills  and  capabilities  managers  view  as  important. 
An  extensive  effort  was  undertaken  in  the  US  at  the  National  Security  Agency  (NSA)  to  identify 
core  competencies  required  of  intelligence  analysts  (see  Moore  and  Krizan  2003,  Moore  2005, 
Moore  et  al.  2005).  Moore  and  Krizan  grouped  the  identified  competencies  for  intelligence 
analysis  into  four  categories  -  abilities,  skills,  knowledge,  and  characteristics  (see  Table  2). 
Krizan  (1999)  further  differentiated  among  four  types  of  intelligence  -  descriptive,  explanatory, 
inteipretive,  and  estimative  intelligence  -  and  Moore  (2005)  suggested  that  a  different  set  of 
capabilities  may  be  required  for  each  type  of  intelligence.  In  our  interviews,  we  did  not  make  a 
distinction  among  different  types  of  analysis  and  specific  skills  required  for  each. 

Moore  and  Krizan’ s  list  of  core  competencies  contains  a  set  of  fairly  general  and  basic  skills  that 
may  overlap  with  essential  capabilities  of  other  professions.  Conducting  an  analysis  of  cross¬ 
functional  skills  and  generalized  work  activities  could  allow  other  jobs  and  job  families  that 
require  similar  capabilities  to  intelligence  analysis  to  be  identified.  Moreover,  identifying  job 
families  related  to  intelligence  analysis  could  provide  an  opportunity  to  leverage  training  and 
assessment  practices  for  certain  skills  shared  with  related  occupations. 

There  is  significant  overlap  between  the  list  of  capabilities  identified  by  the  managers  who  were 
interviewed  ( Table  1)  and  the  list  compiled  by  Moore  and  Krizan  (Table  2).  It  is  worth  noting 
that,  although  the  list  that  we  compiled  based  on  managers’  comments  does  not  include  all  of  the 
items  from  Moore  and  Krizan’ s  list,  our  list  also  includes  capabilities  not  identified  in  the  NSA 
study.  Capabilities  that  were  mentioned  by  participants  in  our  study  that  were  not  described  by 
Moore  and  Krizan  relate  to  analysts’  abilities  to  deal  with  environmental  demands,  such  as 
working  under  stress,  delivering  to  deadlines,  demonstrating  emotional  stability,  dealing  with 
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uncertainty,  making  up  for  missing  information,  making  judgments  under  uncertainty,  being  able 
to  stand  by  their  judgments,  and  possessing  accurate  memory. 

Some  of  the  managers  pointed  out  that  not  just  anybody  could  be  trained  to  be  an  analyst  because 
intelligence  analysis  requires  a  certain  set  of  inherent  abilities.  The  skills  and  capabilities 
mentioned  by  managers  reflect  this  idea;  some  of  the  items  represent  inherent  abilities  that  cannot 
easily  be  learned  or  improved  (e.g.,  thirst  for  knowledge  and  ability  to  cope  with  uncertainty) 
while  other  items  in  the  list  might  be  more  responsive  to  appropriate  training  methods 
(e.g.,  regional  expertise  and  technical  knowledge).  The  managers’  notion  that  intelligence 
analysis  requires  some  inherent  ability  was  also  reflected  in  Moore  and  Krizan’s  results  -  items 
grouped  under  characteristics  and  abilities  represent,  in  the  authors’  opinion,  inherent  qualities, 
while  items  under  knowledge  and  skills  are  viewed  as  those  that  may  be  acquired. 


Table  2:  Moore  and  Krizan’s  list  of  core  competencies  for  intelligence  analysis 


Type 

Items 

Type 

Items 

Abilities 

Communicating  (includes 
aural,  graphic,  visual,  oral) 

Knowledge 

Target  knowledge 

Teaming  and  Collaborating 
(includes  influencing, 
leading,  following, 
synergizing) 

Intelligence  community 

Thinking  (includes 
information  ordering, 
pattern  recognition, 
reasoning-induction, 
deduction,  and  abduction) 

Government  plans  and  policy 

Customers 

Analytic  resources 

Characteristics 

Insatiably  curious 

Skills 

Critical  thinking 

Self-motivated 

Literacy 

Fascinated  by  puzzles 

Computer  literacy 

Exhibits  “AHA”  thinking3 

Expression 

Observes  voraciously 

Foreign  language  proficiency 

Reads  voraciously 

Research 

Fruitfully  obsessed 

Information  gathering  and 
manipulation 

Takes  variable  perspectives 

Project/process  management 

Makes  creative  connections 

3  Note:  “AHA"  thinking  refers  to  the  gaining  of  an  insightful  understanding  of  an  issue,  i.e.,  experiencing  a 
“eureka”  moment 
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Playful 

Has  sense  of  humour 


Has  sense  of  wonder 


Concentrates  intensely 
Questions  convention 


To  identify  core  capabilities  for  intelligence  analysis,  it  may  be  beneficial  to  examine  these 
capabilities  in  light  of  individual  differences  from  research  in  personality  and  social  psychology. 
For  example,  Smith  (2009)  identified  three  factors  of  the  Five  Factor  Model  of  personality  - 
conscientiousness,  agreeableness,  and  openness  to  experience  -  as  important  in  Intelligence 
Officer  training  and  performance.  Girard  (2010)  recommended  including  in  the  selection  process 
the  Trait  Self-Descriptive  Personality  Inventory  Revisited  (TSD-PI)  measure  for  evaluating  the 
relevant  personality  factors.  The  TSD-PI  can  capture  some,  but  not  all  of  the  personality 
characteristics  that  may  be  critical  for  intelligence  analysis.  For  instance,  one  of  the  essential 
activities  in  intelligence  analysis  that  was  identified  is  generation  of  alternatives  and  their  careful 
evaluation  (Davis  2006,  Heuer  1999,  Jervis  2009,  Moore  2007,  Steinberg  2008).  The  ability  to 
effectively  deal  with  uncertainty  and  to  generate  and  thoroughly  evaluate  alternative  explanations 
may  be  contingent  upon  individual  differences  such  as  those  specifically  measured  by  the  Need 
for  Cognition  (NFC)  scale  (Cacioppo  and  Petty  1982)  and  the  Need  for  Cognitive  Closure 
(NFCC)  scale  (Kruglanski  and  Webster  1996),  to  name  but  two  examples. 

The  NFC  scale  is  a  stable  individual  difference  evaluation  that  reflects  the  extent  to  which  an 
individual  seeks  out  and  enjoys  effortful  cognitive  activities.  People  who  score  higher  in  the  NFC 
scale  would  be  more  inclined  to  envision  different  alternatives  and  would  be  more  comfortable 
contemplating  them  under  uncertainty.  The  NFCC  scale  is  another  stable  individual  difference 
measure  that  reflects  one’s  need  for  reaching  closure  or  arriving  at  a  conclusion  on  an  issue  and 
hanging  onto  that  conclusion,  which  provides  a  comfort  of  certainty.  Therefore,  people 
who  score  higher  in  the  NFCC  scale  would  not  tolerate  uncertainty  well,  tend  to  “jump  to 
conclusions,”  be  more  reluctant  to  change  their  minds,  and  be  less  inclined  to  give  all  alternatives 
due  consideration. 

Given  the  nature  of  these  individual  differences  and  the  distinct  requirements  of  intelligence 
analysis,  we  would  expect  more  successful  analysts  to  score  higher  than  average  in  the  NFC  scale 
and  lower  in  the  NFCC  scale.  McLellan  and  Mandel  (2010b)  have  begun  to  assess  Canadian 
intelligence  analyst  trainees  on  these  two  individual  differences  measures,  along  with  a  measures 
of  accuracy  and  coherence  based  on  the  Decision  Making  Competence  index  (de  Bruin  et  al. 
2007).  The  aim  of  their  field  research  is  to  test  whether  or  not  those  individual  differences  predict 
coherence  and  accuracy,  and  to  establish  baseline  levels  for  the  community,  as  compared  to  the 
general  population.  To  the  best  of  our  knowledge,  there  have  been  no  efforts  to  date  to  correlate 
these  individual  differences  with  performance  measures  with  a  sample  of  intelligence  personnel. 
If  proven  to  be  reliable  predictors  of  analysts’  decision  making  performance,  then  the  NFC  and 
NFCC  scales  may  be  usefully  employed  in  the  analyst  selection  process.  The  NFC  and  NFCC 
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scales  may  complement  the  TSD-PI  measure  recommended  by  Girard  (2010)  for  the  selection 
process  of  Intelligence  Officers  and  Intelligence  Operators  in  the  Canadian  Forces. 

The  proposition  that  stable  individual  differences  predict  performance  on  intelligence  analysis 
activities  presents  a  unique  opportunity  for  the  community  to  improve  its  selection  processes. 
Assessment  measures  could  be  developed  to  test  candidates  for  analytic  positions  on  the  requisite 
characteristics.  On  the  other  hand,  acquired  capabilities  could  be  used  for  performance  evaluation. 
However,  more  research  is  required  to  reliably  identify  correlates  of  analytic  performance  before 
such  measures  may  be  implemented. 

3.4  Selection  process 

Results.  The  selection  process  for  intelligence  analysts  is  an  important  issue  for  the 
community  and  some  managers  commented  that  there  is  room  for  improvement  in  the  current 
selection  practices. 

Civilian  analysts  in  both  organizations  are  selected  based  on  their  academic  backgrounds  and 
performance  on  selection  tests.  In  terms  of  their  background,  successful  candidates  are  expected 
to  have  a  Master’s  degree,  usually  in  political  science,  area  studies,  or  history.  A  candidate’s  field 
of  study  is  not  as  restricted  in  CDI,  where  candidates  with  degrees  in  other  disciplines  might  also 
be  considered.  Some  CDI  managers  commented  that  analysts  in  CDI  tend  to  be  generalists  and 
are  usually  moved  among  different  areas  of  analysis;  therefore,  it  may  not  be  as  important  for  new 
analysts  to  have  deep  expertise  in  a  certain  region,  and,  rather,  it  is  the  breadth  of  experience  that 
is  commonly  valued.  Some  managers  from  IAS,  on  the  contrary,  commented  that  their 
organization  is  often  looking  for  specialists  rather  than  generalists,  and  that  it  tends  to  put 
emphasis  on  hiring  people  with  deep  knowledge  of  a  given  area  or  a  region.  Analysts’  knowledge 
and  experience  with  a  given  region  usually  plays  an  important  role  in  the  selection  process  at  IAS. 

In  addition  to  satisfying  background  requirements,  candidates  for  positions  at  both  organizations 
undergo  an  interview  and  a  writing  skills  test,  which  help  managers  to  assess  their  analytic  and 
writing  abilities.  Some  managers  pointed  out  that  after  hiring  an  intelligence  analyst,  it  takes  an 
average  of  six  months  to  one  year  to  be  able  to  truly  assess  their  capabilities  and  fit  for  the  job, 
and,  at  that  point,  it  is  “either  great  or  too  late”  (Interviewee).  Civilian  managers  in  CDI  are  not 
involved  in  the  selection  of  military  intelligence  analysts  that  work  in  CDI;  those  analysts  are 
appointed  to  their  positions  through  a  secondment  process.  The  majority  of  military  analysts  in 
CDI  are  coming  from  the  Intelligence  Branch,  but  some  military  analysts  who  come  from  other 
branches  of  the  CF  might  lack  training  or  background  in  intelligence  analysis. 

There  are  several  factors  that  may  impede  the  selection  process,  four  of  which  are  discussed 
below.  First,  managers  do  not  agree  on  the  precise  skills  and  capabilities  required  of  analysts  to 
perform  their  roles4. 

...  right  now  there  is  not  a  lot  of  it  [selection  based  on  skills].  In  terms  of  being 
selected  as  an  analyst  it  is  just  a  chair  that  we  put  someone  in,  so  there  is  the 
expectation  that  anybody  can  be  an  analyst.  (Interviewee) 


4  There  have  been  recent  community-wide  efforts  to  identify  the  essential  skills  for  intelligence 
professionals.  See,  for  example,  ODNI  (2009). 


48 


DRDC  Toronto  TR  2010-144 


This  issue  not  only  hampers  the  selection  process,  but  also  the  development  of  a  comprehensive 
training  program  for  analysts. 

I  think,  it  will  be  helpful  for  us  as  a  community  if  we  had  a  really  original 
research  project  that  was  looking  at  what  makes  a  good  analyst,  developing  and 
mapping  all  those  skills,  understanding  them  and  being  able  to  teach  them,  and 
then  working  into  how  to  make  a  great  in  telligence  analyst  as  part  of  the  training 
program.  (Interviewee) 

Second,  to  test  candidates  on  the  identified  skills  in  the  selection  process,  accurate  and  effective 
assessment  tools  are  needed. 

There  is  [a  test  to  gauge  capability],  but  it’s  not  very  good.  We’ve  included 
interview  questions  to  get  at  people’s  ability  to  deal  with  large  quantities  of 
information  and  how  they  make  judgments.  But  ultimately  they  are  completely 
inadequate.  People  may  have  interesting  things  to  say,  but  whether  that’s  actually 
how  their  mind  works  when  they  are  sitting  down  here  looking  at  a  problem  may  or 
may  not  be  the  case.  We  are  trying  to  come  up  with  questions  that  will  help  us 
identify  people’s  abilities  in  these  areas  ....  Maybe  a  formal  test  would  be  good,  but 
we  are  not  in  the  position  to  come  up  with  that.  (Interviewee) 

Third,  some  managers  also  commented  that  not  all  skills  and  characteristics  needed  in  successful 
intelligence  analysts  could  be  tested.  They  emphasised  the  individual-environment  fit,  where 
different  organizational  environments  place  unique  demands  on  individuals,  and  it  is  important 
for  individuals  to  be  able  to  cope  with  these  demands. 

The  kind  of  environment  we  are  in  requires  a  level  of  flexibility  -  there  is  a  fair 
bit  of  stress,  a  fair  bit  of  teamwork;  there  are  a  lot  of  outside  demands  ....  I  think 
there  are  a  lot  of  those  kinds  of  factors  that  you  can’t  test  for.  (Interviewee) 

Finally,  the  requirements  of  the  public  service  personnel  selection  process  impose  certain 
constraints  on  the  analyst  selection  process.  Although  most  of  the  managers  commented  on 
deficiencies  of  the  current  selection  practices  in  their  organizations,  they  still  pointed  out 
that  the  process  is  somewhat  effective  and  allows  them  to  select  capable  individuals  with 
reasonable  success.  However,  it  is  unknown  whether  or  not  the  process  selects  those  best-fitted 
for  the  job  and  whether  or  not  other,  more  capable  individuals  were  screened  out  by  the  current 
selection  processes. 

Discussion.  Several  factors  constrain  the  analyst  selection  process.  First,  the  ability  to  identify  the 
essential  competencies  required  from  potential  recruits  is  one  of  the  constraints,  and  is  directly 
related  to  our  discussion  in  the  section  “Essential  Skills”  above.  As  Moore  (2005)  suggested,  core 
competencies  required  of  analysts  may  differ  from  one  organization  to  another,  and  even 
for  different  positions  within  the  same  organization.  The  requisite  competencies  for  a 
position  are  determined  by  the  types  of  activities  an  analyst  is  expected  to  carry  out,  and  the 
demands  of  the  organization. 

Second,  there  is  a  need  for  tools  to  effectively  assess  the  identified  competencies.  Based  on  the 
interviewees’  comments,  there  seem  to  be  a  need  for  both  -  identification  of  competencies  and 
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tools  for  their  assessment.  A  third  constraint  according  to  the  managers  is  imposed  by  uniform 
organizational  policies,  which  govern  the  selection  process  and  limit  its  flexibility. 

A  fourth  and  final  constraint  in  the  selection  process  is  the  lack  of  professional  training  and 
education  in  the  intelligence  community.  There  are  no  special  training  programs  to  prepare 
individuals  entering  the  intelligence  analysis  profession,  and  training  occurs  only  after  analysts 
have  been  hired  into  their  positions.  A  professional  education  program  in  intelligence  analysis 
prior  to  entering  the  job  market,  supplemented  with  a  university  co-operative  program,  may  be 
beneficial  to  both  students  and  intelligence  organizations.  Co-operative  education  programs  allow 
students  to  supplement  classroom  study  with  relevant  work  experience  in  varied  contexts  to  better 
prepare  them  for  the  challenges  of  their  future  careers.  On  the  other  hand,  intelligence 
organizations  may  be  able  to  better  evaluate  potential  candidates  by  observing  students  in  a  work 
context.  The  feasibility  and  desirability  of  implementing  this  training  program  depends  on  the 
state  of  professionalization  of  the  intelligence  community  and  on  the  demand  for  such 
professionals,  especially  given  the  relatively  small  size  of  the  Canadian  community. 

The  intelligence  analysis  selection  process  may  benefit  from  further  research  to  identify  essential 
capabilities  for  intelligence  analysis  as  well  as  to  develop  evaluation  tools  to  measure  these 
capabilities.  In  addition,  as  suggested  above,  an  alternative  or  a  supplementary  approach  to 
candidate  evaluation  may  be  achieved  through  internship  and  co-operative  education  programs 
that  will  bring  prospective  candidates  into  the  organization’s  work  environment  for  a  prolonged 
period  of  time,  allowing  managers  to  observe  and  evaluate  prospective  candidates  in  a  realistic 
work  environment. 

3.4.1  Manager  selection:  Analyst-to-manager  progression 

Results.  CDI  managers  commented  that  intelligence  analysis  is  seen  as  a  junior  activity  in  the 
military  community.  Military  analysts  do  not  stay  in  their  analytic  positions  for  prolonged 
periods,  partially  due  to  the  rotational  nature  of  military  posting  cycles  and  partially  due  to  the 
promotion  process.  Successful  military  analysts  are  expected  to  move  out  of  analytic  roles  and 
move  up  to  capability  management.  This  creates  a  somewhat  paradoxical  situation,  in  which  good 
analysts  do  not  stay  in  their  positions  long  because  they  are  promoted.  The  resulting  rotation  of 
personnel  through  analytic  roles  prevents  continuity  and  expertise  retention  in  the  roles. 

Personnel  turnover  is  not  as  much  of  an  issue  in  the  civilian  sector,  where  analysts  have  an 
opportunity  to  stay  in  analytic  roles  throughout  their  career.  As  in  the  military,  however,  the  best 
analysts  move  out  from  analytic  roles  to  management  positions,  where  some  struggle  to  make 
time  to  engage  in  analytic  work  while  others  abandon  analysis  altogether.  Some  managers 
commented  that  the  skill  set  required  for  intelligence  analysis  is  quite  different  from  that  needed 
to  be  an  effective  manager;  good  analysts  do  not  necessarily  make  good  managers. 

Discussion.  Identification  of  managerial  candidates  and  succession  management  are  important 
issues  for  the  intelligence  community.  Intelligence  managers  not  only  play  a  significant  role  in 
shaping  the  analytical  process  (Moore  et  al.  2005),  they  also  significantly  influence  the 
professional  development  of  intelligence  analysts  reporting  to  them.  As  we  discuss  in  the 
following  section,  development  of  intelligence  analysts  in  the  Canadian  community  greatly  relies 
on  a  mentoring  model  in  which  an  analyst’s  immediate  manager  provides  mentorship  to  the 
analyst.  Thus,  it  is  crucial  to  the  development  of  junior  analysts  that  their  managers  have 
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knowledge  and  experience  in  intelligence  analysis.  As  some  managers  have  pointed  out,  however, 
excelling  at  analysis  is  not  sufficient  to  be  a  successful  manager.  Because  the  skill  sets  required  to 
be  a  successful  analyst  differ  from  those  of  managers,  it  is  important  to  identify  the  distinct 
capabilities  required  to  be  an  effective  manager  in  the  context  of  intelligence  analysis. 
Unfortunately,  we  did  not  discuss  in  detail  the  promotion  and  succession  process  and  practices 
during  our  interviews  with  the  managers.  There  is  a  dearth  of  literature  on  this  topic  as  it  relates  to 
the  intelligence  community,  an  exception  being  a  report  by  Hatfield  (2008)  that  deals  with  issues 
of  succession  management  in  the  top  three  tiers  of  the  US  intelligence  community. 

3.5  Training 

3.5.1  Current  practices 

Results.  Some  managers  commented  that  new  intelligence  analysts  get  most  of  their  analytic 
training  after  they  join  the  organization  through  mentoring  from  their  managers  and  introductory 
courses  organized  by  individual  organizations,  as  no  specific  training  in  intelligence  analysis  is 
available  in  Canada  prior  to  joining  one  of  the  intelligence  organizations. 

All  the  intelligence  analysis  training  the  analysts  get  on  the  job.  (Interviewee) 

For  the  first  6  months  on  the  job,  analysts  typically  learn  through  a  mentorship 
relationship  and  interaction  with  their  directors.  (Interviewee) 

Some  managers  also  pointed  out  that  the  quality  of  the  mentoring  relationship  is  more  important 
for  improving  analysis  than  various  analytic  tools  and  procedures. 

Different  organizations  have  their  own  internal  training  courses  tailored  to  suit  their  specific 
needs.  An  example  of  such  training  is  IAS’s  internal  six-day  introductory  course  in 
intelligence  analysis,  which  is  offered  approximately  every  two  years,  depending  on  the  number 
of  new  analysts. 

We  did  not  discuss  military  intelligence  training  in  detail,  but  managers  at  CDI  pointed  out  that 
there  is  a  much  more  structured  training  program  for  the  military  intelligence 
professionals  that,  among  other  aspects  of  the  intelligence  trade,  also  include  components  on 
intelligence  analysis.  Civilian  analysts  normally  do  not  have  access  to  the  analytic  training 
provided  to  military  analysts. 

Discussion.  Analytic  training,  at  present,  occurs  mainly  in  two  ways:  through  mentoring  and 
through  training  courses.  There  are,  no  doubt,  advantages  to  mentoring,  such  as  individual 
attention  in  addressing  the  specific  needs  of  each  analyst  at  a  comfortable  pace.  However,  as  there 
is  no  agreed  upon  approach  to  analysis,  it  is  likely  that  different  managers  emphasise  different 
methods  and  approaches  to  analysis,  and  managers  may  have  varying  requirements  and  standards. 
An  individual  manager’s  expertise  in  analysis  and  his  or  her  mentoring  skills  are  important  to  the 
effectiveness  of  mentoring.  Moreover,  the  mentoring  approach  to  training  is  very  time 
consuming  for  managers,  which  may  be  problematic,  especially  if  the  number  of  supervised 
analysts  increases. 
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In  addition  to  receiving  mentoring,  analysts  may  be  able  to  partake  in  various  training  modules 
that  may  be  offered  through  their  own  organizations,  through  the  PCO  or  other  Canadian 
departments,  or  through  programs  offered  by  allied  countries.  These  modules  address  particular 
topics  that  are  largely  independent  of  each  other.  Whether  or  not  analysts  are  able  to  take 
advantage  of  these  training  courses  greatly  depends  on  the  needs,  current  pressures,  and  resources 
of  their  organizations. 

Some  agencies  organize  and  deliver  their  own  in-house  training  courses  for  analysts.  Organizing 
and  maintaining  such  internal  training  programs  is  very  time-  and  resource-consuming  for 
organizations  and  significantly  limits  what  a  single  organization  may  be  able  to  do.  The 
establishment  of  centralized  community-wide  training  in  intelligence  analysis  may  be  more  cost 
effective,  ensure  uniformity  of  training  across  the  community,  and  promote  the  development  of 
community-wide  analytic  standards.  Reviewing  US  IC  reform,  Hulnick  (2008)  emphasised  the 
importance  of  centralized  basic  training  for  intelligence  professionals.  In  Hulnick’ s  opinion, 
centralized  training  would  reduce  costs  and  provide  an  important  opportunity  for  analysts  to  build 
relationships  and  professional  networks  with  peers  from  other  agencies  that  would  promote 
information  sharing  and  cross-agency  collaboration. 

Treverton  and  Gabbard  (2008)  observed  that  a  community-wide  course  in  intelligence  analysis  is 
lacking  and,  like  analysts  in  Canada,  analysts  in  the  US  receive  most  of  their  training  on  the  job. 

In  the  Canadian  IC,  the  Intelligence  Assessment  Learning  Program  (IALP)  was  established  to 
oversee,  develop  and  coordinate  training  efforts  in  the  community.  The  PCO,  through  the  IALP, 
has  developed  a  ten-day  community-wide  introductory  intelligence  analysis  course.  The  course 
covers  foundations  of  intelligence  analysis,  and  has  been  attended  by  analysts  from  various 
departments  since  2006  (Pyrik  2007).  There  are  also  seminars  and  shorter  modules  offered 
through  the  IALP  to  address  specific  issues.  The  introductory  IALP  course  is  not  mandatory  for 
analysts  and  not  all  agencies  enrol  their  analysts  in  this  course.  Some  managers  echoed  the 
opinion  of  Hulnick  (2008)  that  one  of  the  benefits  of  attending  a  community-wide  course  is 
meeting  people  from  other  departments  and  expanding  the  analysts’  professional  network. 

At  present,  training  for  intelligence  analysts  is  rather  haphazard,  and  analyst  development  lacks  a 
unifying  structure  that  would  ensure  a  well  rounded,  systematic,  extensive  and  comprehensive 
education  in  intelligence  analysis  practices.  Establishing  such  a  unifying  training  structure  for 
analysts  would  require  first  defining  the  discipline  of  intelligence  analysis  and  the  core 
competencies  that  have  begun  to  emerge  in  the  community  literature  (Bruce  and  George  2008, 
Johnston  2005,  Marrin  and  Clemente  2006).  A  professional  development  curriculum  would  rely 
on  identified  core  competencies  for  intelligence  analysis  and  would  be  informed  by  training 
experiences  of  Canada’s  allies  (Goodman  and  Omand  2008,  Marrin  2003).  The  development  of  a 
community-wide  training  program  could  foster  the  development  of  professional  performance 
standards  and  personnel  management  practices,  which  in  turn  may  advance  the 
professionalization  of  intelligence  analysis.  However,  the  need  and  feasibility  of  such  a  program 
in  the  Canadian  context  would  need  to  be  determined  by  the  community,  and  would  depend  on 
the  resources  available. 

Intelligence  training  is  also  distinct  from  what  might  be  called  intelligence  education.  Whereas 
training  usually  focuses  on  teaching  trainees  a  certain  set  of  techniques  or  specific  skills, 
education  often  focuses  on  developing  a  critical  mindset  -  one  that  values  intellectual  curiosity, 
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creativity,  and  other  liberal  values.  In  the  intelligence  domain,  the  question  of  education  hinges  to 
a  large  extent  on  whether  one  views  intelligence  analysis  more  as  a  profession  or  as  a  trade. 
Given  the  difficulties  that  the  IC  has  in  handling  training,  one  might  expect  the  education  of 
future  analysts  to  fall  under  the  remit  of  universities.  Although  a  number  of  Canadian  universities 
offer  various  courses  on  intelligence  subjects,  the  international  affairs  program  at  Carleton 
University,  Ottawa,  Ontario,  is,  perhaps,  the  only  civilian  university  program  in  Canada  that 
incorporates  intelligence  studies  within  its  international  studies  program  (Rudner  2009).  Rudner 
(2009)  outlined  a  number  of  reasons  for  the  lack  of  intelligence  programs  in  higher  education.  On 
the  one  hand,  secrecy  associated  with  intelligence  activities  and  their  clandestine  nature  may  have 
led  to  reluctance  on  the  part  of  universities  and  academics  to  engage  in  teaching  and  research  in 
the  area  of  intelligence.  On  the  other  hand,  inadequate  public  awareness  may  have  resulted  in  the 
lack  of  government  funding  to  support  academic  research  in  the  intelligence  domain  that  would 
have  not  only  contributed  to  the  development  of  the  area,  but  also  prepared  qualified  faculty.  A 
number  of  Canadian  government  organizations  employ  civilian  intelligence  analysts  (among  them 
are  CDI,  PCO,  Canadian  Security  Intelligence  Services,  Royal  Canadian  Mounted  Police,  Canada 
Border  Services  Agency,  Transport  Canada,  Environment  Canada,  Citizenship  and  Immigration 
Canada),  and  specialized  programs  and  individual  courses  in  Intelligence  Studies  go  a  long  way 
towards  developing  students  for  a  career  as  intelligence  analysts.  Rudner  (2009)  emphasised  the 
importance  of  developing  such  programs  and  establishing  centers  of  Intelligence  Studies  that 
would  be  proportionate  to  the  national  interest  in  intelligence  affairs.  Moreover,  civilian 
university  or  college  training  in  intelligence  analysis  that  could  fully  prepare  potential  analysts  to 
work  in  an  intelligence  agency  is  even  more  limited.  This  is  unsurprising  because  intelligence 
agency  work  requires  a  certain  level  of  classification  that  a  university  environment  would  be 
unable  to  provide,  and,  hence,  most  of  the  training  in  analytic  tradecraft  analysts  receive  occurs 
after  joining  an  agency,  through  working  closely  with  managers,  as  noted  earlier. 

3.5.2  Training  challenges 

3.5.2. 1  Consensus  regarding  core  competencies  for  intelligence  analysis 

Results.  One  of  the  biggest  challenges  in  training  intelligence  analysts  is  the  lack  of  available 
systematic  training  programs  that  focus  on  analysis.  Some  managers  commented  that  the  work 
that  the  PCO  has  done  in  developing  community-wide  intelligence  training  is  a  great  step 
forward,  but  much  is  left  to  be  done.  One  of  the  difficulties  in  developing  such  a  program, 
according  to  the  managers,  is  reaching  an  agreement  within  the  community  on  “exactly 
what  it  takes  to  make  a  great  intelligence  analyst,”  that  is,  what  are  the  core  competencies  for 
intelligence  analysis. 

Training  is  an  important  issue  that  the  intelligence  community  has  tried  to  deal 
with  for  some  time.  It  is  not  easy  because  the  requirements  are  somewhat 
different  in  each  department.  (Interviewee) 

When  you  speak  of  core  competencies,  differen  t  agencies  are  going  to  understand 
different  definitions  for  core  competencies  ...  when  we  first  looked  at  the  training 
program  we  wanted  to  look  at  what  were  the  core  competencies  of  analysts  so 
that  we  could  tailor  the  training  program.  That  was  like  pulling  teeth!  ...  getting 
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the  community  to  submit  to  us  their  competencies  was  incredibly  tedious. 

(Interviewee) 

Discussion.  A  community-wide  training  program  for  intelligence  analysts  would  need  to  hone 
knowledge  and  skills  that  are  valuable  and  pertinent  to  various  agencies  within  the  community. 
Therefore,  agreement  within  the  community  on  what  such  a  program  should  emphasize  is 
important  to  ensure  acceptance  of  the  program.  As  different  intelligence  departments  are  faced 
with  a  myriad  of  issues  and  pressures,  they  have  different  conceptions  of  core  competencies  for 
intelligence  analysts.  However,  the  reason  for  difficulty  in  reaching  a  consensus  may  be  two-fold: 
it  is  difficult  for  departments  to  identify  core  competencies  pertinent  to  their  organization,  and 
there  exist  considerable  inter-departmental  differences  in  the  sets  of  competencies  identified.  The 
former  issue  may  require  internal  focussed  efforts  in  analysing  a  department’s  requirements5, 
while  the  latter  may  require  inter-departmental  negotiation  and  a  broader  view  of  the  tradecraft  of 
intelligence  analysis. 

Notwithstanding  inter-organizational  disagreements  on  what  constitutes  analytic  competencies, 
consensus  may  be  possible.  Krizan  (1999)  differentiated  among  four  levels  of  intelligence 
analysis  -  descriptive,  explanatory,  interpretive,  and  estimative.  Moore  (2005),  using  Krizan’s 
typology  of  analysis  and  the  set  of  core  competencies  of  Moore  and  Krizan  (2003)  for  intelligence 
analysis,  argued  that  each  type  of  analysis  may  require  a  different  set  of  abilities,  skills,  and 
knowledge.  Furthermore,  Moore’s  analysis  revealed  that  despite  the  differences  among  the  four 
types  of  analysis,  there  remains  a  common  set  of  abilities,  skills,  and  knowledge  required  for  all: 
information  gathering  and  manipulation  skills,  critical  reasoning,  computer  literacy,  oral  and 
written  communication  skills,  research  skills,  knowledge  of  the  intelligence  community,  and 
knowledge  of  government  plans  and  policies6. 

The  extent  to  which  Moore  and  Krizan’s  set  of  core  competencies  for  intelligence  analysis  is 
pertinent  to  the  Canadian  community  remains  to  be  determined,  as  well  as  the  extent  to  which 
Krizan’s  typology  of  analytic  tasks  reflect  those  faced  by  Canadian  agencies.  However,  Moore’s 
analysis  is  encouraging  as  it  suggests  that  despite  differences  among  organizations,  it  may  be 
possible  to  identify  a  common  set  of  core  competencies  required  of  all  intelligence  analysts. 
Hulnick  (2008),  emphasising  the  importance  of  community -wide  basic  training  for  intelligence 
analysts,  noted  that  inter-departmental  differences  are  inevitable  and  that  the  need  for  focused 
agency-specific  training  would  remain.  This  may  be  the  case  for  the  Canadian  intelligence 
community  as  well. 


3.5.2. 2  Assessing  training  effectiveness 

Results.  Another  challenge  in  training  intelligence  analysts  is  assessing  overall  effectiveness  of 
training  in  the  required  competencies. 


5  Such  as  the  DGMPRA’s  efforts  in  conducting  the  Intelligence  Officer  job  analysis  and  identifying  core 
competencies  for  Intelligence  Officer  and  Intelligence  Operator  occupations  (Girard  2010,  Smith  2009). 

6  For  more  information,  see  Moore  (2005). 
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Clear  measures  to  evaluate  effectiveness  of  the  training  ( i. e. ,  improvements  in 
cognitive  ability  and  thinking  skills)  are  also  needed.  However,  assessing 
analytical  products  is  a  highly  subjective  process.  (Interviewee) 

Discussion.  Several  dimensions  on  which  a  training  program  could  be  evaluated  have  been 
identified  in  the  training  literature:  trainees’  reactions  to  training,  actual  learning,  changes  in  the 
workplace  behaviour  resulting  from  the  transfer  of  training  (i.e.,  the  application  of  learned 
principles  to  the  practical  setting),  improvement  in  performance  or  organizational  results 
(Kirkpatrick  1967,  1979),  and  return  on  investment  (ROI)  (Phillips  1996).  The  last  two  criteria 
(i.e.,  organizational  results  and  ROI)  are  perhaps  the  most  significant  factors  in  determining  the 
rationale  for,  and  effectiveness  of,  training  from  an  organizational  perspective.  An  effective 
training  program  ought  not  only  to  ensure  that  trainees  acquire  and  apply  new  knowledge  and 
skills,  it  should  also  ensure  that  the  application  of  training  leads  to  measurable  performance 
improvement  (i.e.,  the  organizational  results).  Investment  of  organizational  resources  into  a 
training  program,  even  if  it  is  only  analysts’  time  away  from  their  regular  duties,  could  only  be 
justified  if  the  benefits  resulting  from  participation  or  administration  of  the  training  program 
outweigh  the  costs. 

Identifying  a  set  of  essential  capabilities  for  intelligence  analysts  that  correlates  with  their 
performance  may  be  instrumental  to  the  development  and  evaluation  of  training  programs. 
Changes  in  analysts’  competencies  due  to  training  could  be  assessed  directly  through,  for 
example,  administration  of  a  standardized  test  of  the  target  capability  before  and  after  the 
training.  However,  measuring  the  extent  to  which  improvements  in  the  target  capability  transfer 
to  behavioural  changes  in  the  workplace  or  have  an  impact  on  analyst’s  performance  is  not  a 
straightforward  task.  It  has  been  recognized  in  the  training  literature  that  measuring 
organizational  results  emerging  from  training  or  its  costs  and  benefits  is  considerably  more 
difficult  than  measuring  trainees’  reactions  to  training  or  their  learning  (Blanchard  et  al.  2000). 
The  difficulty  of  evaluating  organizational  results  of  training  in  the  intelligence  analysis 
context  is  also  amplified  by  the  subjective  nature  of  product  and  performance  evaluation 
prevailing  in  the  community.  In  the  initial  stages,  evaluation  of  analytic  training  may  have  to  rely 
on  objective  measures  of  learning  and  managers’  and  analysts’  own  subjective  evaluations  of  the 
training  impact. 

Intelligence  organizations  require  training  programs  that  improve  analysts’  task-related 
capabilities  and  allow  them  to  be  more  effective  in  their  jobs.  In  other  words,  the  ultimate  goal  of 
analytic  training  programs  is  to  enable  analysts  to  become  better  analysts.  Determining  the 
success  of  a  training  program  is  tightly  intertwined  with  the  identification  of  core  competencies, 
measures  of  analytic  performance,  and  quality  of  analytic  products.  That  is,  one’s  evaluation  of 
the  effectiveness  of  a  training  program  depends  on  how  one  defines  effective  analysis.  Along  with 
the  development  of  a  structured  intelligence  analysis  training  program,  measures  need  to  be 
developed  and  agreed  upon  to  assess  whether  or  not  the  training  program  achieves  its  goals. 
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3.5.2. 3  Transfer  of  training:  the  role  of  the  favourable  workplace  environment 


Results.  Some  managers  recognized  that  training,  however  effective  it  may  be  in  the  short  term, 
may  still  be  ineffective  in  changing  analysts’  behaviour  if  the  trained  principles  are  not  reinforced 
on  the  job. 

My  problem  is,  once  someone  has  finished  the  course  and  goes  back  to  their 
regular  division,  unless  others  are  taking  a  similar  approach,  a  lot  of  it  falls  by 
the  wayside.  (Interviewee) 

Discussion.  Training  success  depends  on  a  variety  of  factors.  First,  performance  requirements  are 
identified  that  are  to  be  attained  by  the  training.  Second,  the  capability  that  will  result  in  the 
identified  performance  requirements  is  identified.  Third,  the  training  program  is  designed  to 
ensure  improvement  in  the  identified  capability.  Fourth,  trainees  need  to  be  able  and  motivated  to 
learn  and  later  transfer  the  new  knowledge  to  the  workplace.  Finally,  a  favourable  environment 
needs  to  be  created  to  ensure  training  transfer  and  sustainable  application.  It  has  been  proposed  in 
the  training  literature  that  training  transfer  is  affected  by  a  system  of  influences  including 
trainees’  characteristics,  training  design,  and  work  environment  (Baldwin  and  Ford  1988,  Holton 
et  al.  2000).  Each  of  these  three  factors  is  crucial  to  ensuring  training  effectiveness,  but  it  is  the 
latter  issue,  the  work  environment,  that  is  discussed  presently. 

Intelligence  analysis  is  conducted  in  a  particular  organizational  context  with  specific  situational 
demands  and  constraints  placed  on  analysts.  In  addition,  each  manager  has  his  or  her  own 
requirements  and  standards  for  analytic  products,  which  analysts  have  to  satisfy.  A  training 
program  on  analytic  tradecraft  aims  to  change  the  way  an  analyst  conducts  various  aspects  of 
intelligence  production,  or,  in  other  words,  the  aim  of  training  is  to  change  analysts’  behaviour  in 
a  way  that  improves  performance.  Success  and  effectiveness  of  a  training  program  depend  on 
whether  or  not  the  training  produces  long  lasting  changes  in  behaviour  and  stability  of  the  learned 
principles,  techniques,  and  skills  over  time.  However,  a  training  program  that  results  in  learning 
essential  job  capabilities  is  not  sufficient  for  ensuring  that  trainees  use  what  they  have  learned  in 
the  training  in  their  day-to-day  jobs.  One  of  the  essential  components  in  ensuring  long  lasting 
change  in  behaviour,  in  addition  to  an  effective  training  program  and  willing  and  able  trainees,  is 
creating  environmental  conditions  that  provide  an  opportunity  to  utilize  learned  principles  and 
reinforce  their  application,  thus  creating  an  appropriate  behavioural  setting  (Barker  1968).  The 
work  environment  has  been  shown  to  have  a  significant  impact  on  whether  or  not  trainees  use  the 
newly  acquired  capability  in  the  workplace  (Mathieu  et  al.  1992,  Mathieu  et  al.  1993,  Tracey  et 
al.  1995).  Some  managers  realized  that  if  the  environment  does  not  encourage  application  of  new 
skills,  then  these  skills  have  a  greater  chance  of  not  being  utilized  in  practice. 

For  a  training  program  to  be  effective,  it  is  not  sufficient  to  design  a  comprehensive  curriculum 
and  to  have  all  analysts  participate  in  the  program.  The  situational  demands  and  constraints  that 
are  placed  on  analysts  must  also  be  adjusted  to  ensure  that  analysts  have  the  opportunity  to 
incorporate  the  learned  principles  into  their  analytic  practice.  This  includes  a  need  for  managers 
to  acknowledge  the  importance  of  the  principles  that  were  taught,  and  to  temporarily  adjust  their 
requirements  from  analysts.  Individuals  are  more  likely  to  relapse  to  their  old  way  of  doing  things 
when  they  are  put  under  pressure  (Zakay  and  Wooler  1984).  Thus,  a  temporarily  reduced 
workload  for  analysts  may  allow  them  to  internalize  the  new  approaches  and  make  them  a  part  of 
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their  practice.  Workload  reduction  may  be  particularly  difficult  to  implement  due  to  limited 
resources  and  pressing  timelines. 

3.6  Analytic  Tools 

Results.  During  the  interviews  different  managers  mentioned  a  variety  of  structured  analytic  tools 
that  analysts  may  potentially  use  in  their  work.  Some  of  these  include:  externalization,  mapping, 
devil’s  advocate,  red  cell  analysis,  decision  trees,  link  analysis,  cluster  analysis,  lists,  and  ACH. 
Managers  pointed  out  that  analysts  are  aware  and  familiar  with  (at  least  some  of  the)  tools. 
However,  managers  from  both  organizations  commented  that  analytic  tools  are  not  being  used  on 
a  regular  basis,  and  some  managers  pointed  out  that  there  is  resistance  to  the  application  of  tools 
within  the  community.  This  resistance  was  attributed  to  the  fact  that  application  of  analytic  tools 
might  conflict  with  the  currently  dominant  intuitive  approach  to  analysis. 

There  is  a  hesitancy  to  use  those  kinds  of  things  [structured  analytic  tools] 
because  essentially  the  approach  to  analysis  across  the  community  is  the  intuitive 
approach  ...  .  So  any  kind  of  tool  is  seen  with  a  little  bit  of  suspicion.  Some  tools 
are  too  complex  and  can  take  too  much  time.  There  are  some  very  simple  tools 
that  should  be  used  more  frequently  like  just  making  lists  and  things  like  that .... 

I  don’t  even  think  those  are  used  often  either  ...  .  There  is  resistance  because 
there  is  no  magic  tool  that  you  can  use  to  get  the  right  answer.  (Interviewee) 

Some  managers  also  noted  that  application  of  analytic  techniques  requires  additional  time 
investment  on  the  part  of  analysts.  In  addition,  expertise  in  training  and  support  for  some  of  the 
tools  and  techniques  is  not  available  within  the  Canadian  IC. 

All  of  them  [structured  analytic  tools]  I’m  sure  have  value,  but  they  also  take  up 
a  certain  amount  of  time,  and  there  is  a  certain  amount  of  experience  required  to 
make  them  fully  effective,  and  we  don’t  have  that  experience  here  in  Canada. 
(Interviewee) 

Other  barriers  to  the  application  of  analytic  tools  are  specific  organizational  constraints  such  as 
short  timelines  for  turning  around  an  assessment  and  a  limited  number  of  available  analysts. 

We  teach  a  lot  of  tools...  The  only  downside  with  those  tools  is  that  they  are  most 
effectively  used  when  you  have  the  resources  necessary  to  give  the  time  required 
to  let  someone  do  that.  None  of  those  tools  can  be  used  effectively  in  a 
compressed  period  of  time  ....  There  is  a  portfolio  of  analytic  tradecraft  tools 
that  analysts  are  aware  of  but  in  practice,  given  the  time  constraints,  often  the 
reality  is  that  they  don ’t  have  the  time  to  learn  and  use  them.  (Interviewee) 

Discussion.  Recent  reviews  of  the  intelligence  community  (e.g.,  9/1 1  Commission  2004,  Butler  et 
al.  2004)  called  for  changes  and  reorganization  of  intelligence  activities.  The  recent  trend  in  the 
US  IC  has  been  on  emphasising  the  development  and  application  of  various  analytic  tools  and 
techniques  that  are  purported  to  help  manage  information  overload  and  to  reduce  the  impact  of 
cognitive  biases  in  analysis.  As  a  result,  over  the  years,  various  analytic  tools  and  techniques  have 
been  adapted  to,  or  developed  for,  intelligence  analysis  with  the  aim  to  improve  human 
performance  on  analytic  tasks.  Some  of  these  tools  are  described  in  the  literature  and  training 
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manuals,  and  are  supported  by  software  packages  available  for  intelligence  analysts  to  use  in  their 
assessments  (CIA  2005,  2009,  DIA  2008,  Heuer  2008,  Heuer  and  Pherson  2010). 

Some  managers  that  we  interviewed  acknowledged  that  the  current  trend  in  the  study  of  analytic 
tradecraft  is  in  the  development  of  new  tools  “ that  are  designed  to  mimic  the  human  reasoning 
process.  ”  They  warned  against  this  and  stressed  instead  the  importance  of  “ reflection  on  the 
theoretical  basis  and  limitations  of  intelligence  analysis.  ”  Moore  (2007)  pointed  out  that 
“analysts  and  analysts  alone  create  intelligence.  Although  technological  marvels  assist  analysts  by 
cataloguing  and  presenting  data,  information  and  evidence  in  new  ways,  they  do  not  do  analysis” 
(p.l).  In  the  opinion  of  the  managers  interviewed,  tools  need  to  be  simple  -  they  should  help 
analysts  manage  information,  externalize  the  argumentation  process,  and  to  facilitate  discussion 
with  other  analysts,  while  minimizing  the  time  and  cognitive  effort  required  by  the  analyst  to 
focus  on  learning  how  to  use  the  tool  in  the  first  place  and  on  how  to  implement  it  in  each 
instance  of  application. 

Many  of  the  analytic  tools  that  have  been  developed  and  applied,  however,  have  not  been 
systematically  evaluated.  Although  each  makes  intuitive  sense,  few  studies  have  actually  been 
conducted  to  determine  whether  or  not  their  application  leads  to  the  expected  type  of 
improvement  in  judgment.  In  fact,  the  literature  on  analytic  technique  evaluation  is  sparse  and 
fragmented  (Johnston  2005).  Exceptions  include  work  by  Folker  (2000),  Cheikes  et  al.  (2004)  and 
Pirolli  (2006)  that  evaluated  ACH  as  developed  by  Heuer  (1999). 

If  intelligence  analysis  undergoes  professionalization,  it  will  potentially  bring  about  the 
development  and  formalization  of  best  practices  and  standards  (Fisher  and  Johnston  2008). 
However,  adoption  of  certain  tools  and  techniques  as  process  standards  in  the  intelligence 
community  first  requires  their  systematic  evaluation,  which  has  not  yet  been  done,  or,  at  least,  it 
is  not  evident  in  the  literature.  A  certain  degree  of  reluctance  from  the  community  in  adopting 
these  techniques  may  be  due  to  the  lack  of  empirical  support  for  their  effectiveness.  An  additional 
obstacle  to  the  adoption  of  tools  in  the  community  is  the  lack  of  resources  available  for  training, 
implementation,  and  support  of  these  tools.  The  plethora  of  available  tools  and  uncertainty 
regarding  their  validity  makes  investing  scarce  resources  into  developing  expertise  in  these  tools 
hard  to  justify. 

Systematic  evaluation  of  analytic  tools  is  fundamental  for  advancing  intelligence  analysis  as  a 
profession.  If  done  carefully  using  behavioural  science  research  methods,  it  ought  to  yield  a 
superior  understanding  of  the  benefits  and  drawbacks  of  various  tools  to  support  analysis. 

3.7  Managers’  roles  and  challenges 

During  our  interviews,  some  managers  also  commented  on  their  own  roles  and  the  challenges 
they  face.  We  did  not,  however,  spend  much  time  discussing  these  issues  because  our  primary 
concern  was  on  understanding  issues  related  to  intelligence  analysts. 
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3.7.1  Manager’s  role 

Results.  Managers  listed  a  number  of  responsibilities  that  they  have,  including: 

•  Generating  the  production  outlook  for  their  group;  this  includes  defining  the  scope  and 
framing  the  questions  for  analytic  products. 

•  Establishing  and  maintaining  an  interface  with  clients,  that  is,  the  consumers  of  intelligence 
products.  This  is  especially  important  for  defining  priorities  for  assessments  and  seeking 
feedback  on  final  products. 

•  Actively  participating  in  the  intelligence  community  through,  for  example,  chairing  an  IEG. 

•  Ensuring  production  and  timely  delivery  of  analytic  products  of  satisfactory  quality. 

•  Ensuring  analytic  rigour  through  monitoring  analytic  processes  and  challenging  analysts  in 
their  assumptions,  process,  and  judgments.  As  mentioned  above  in  Section  3.2,  the 
challenge  function  can  be  difficult  for  both  analysts  and  their  managers.  It  requires 
deep  knowledge  of  analytic  processes,  solid  area  knowledge,  and  a  great  deal  of  tact 
from  managers. 

•  Mentoring  and  developing  (new)  intelligence  analysts. 

•  Evaluating  the  quality  of  analytic  products  and  analysts’  performance. 

•  Being  involved  in  the  analysts’  promotion  process. 

•  Fulfilling  financial  management  duties  and  managing  budgets. 

•  Scheduling  and  allocating  resources. 

•  Providing  opportunities  for  professional  development  for  their  analysts. 

•  Being  good  listeners  in  attending  to  analysts’  concerns. 

•  Shielding  their  group  from  various  external  pressures  as  much  as  possible  to  allow  them  to 
concentrate  on  doing  analysis. 

Discussion.  Managers  are  an  essential  part  of  analysts’  environments,  and  they  are  involved  in 
structuring,  monitoring,  and  evaluating  the  analytic  process.  In  other  words,  managers  shape  the 
environment  in  which  intelligence  production  occurs.  Indeed,  in  Moore  and  Krizan’s  words 
“management  has  a  role  to  play  in  making  the  analysis  process  successful”  (p.2 1 1 ,  2005).  As  can 
be  seen  from  the  above  list,  intelligence  managers  perform  a  combination  of  planning,  directing 
or  motivating,  organizing,  liaising,  and  controlling  functions,  not  unlike  managers  in  other  sectors 
of  the  public  service  and  industry.  Communication  with  subordinates,  superiors,  and  external 
parties  constitutes  a  large  part  of  managers’  activities.  In  addition  to  these  tasks,  intelligence 
managers  provide  individual-based  training  and  mentoring  of  new  analysts.  Some  managers 
actively  participate  in  the  inter-organizational,  community-wide  training  of  analysts  and 
community  efforts  directed  towards  professionalization  of  intelligence  analysis. 

As  we  discussed  in  Sub-section  3.1.9,  the  intelligence  organizations  examined  in  the  current  study 
rely  largely  on  the  coordination  of  intelligence  activities  through  direct  supervision  (Mintzberg 
1979),  which  requires  close  personal  contact  between  managers  and  analysts.  The  requirements  of 
this  coordination  mechanism  limit  the  number  of  analysts  any  one  manager  can  effectively 
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supervise  (i.e.,  manager’s  span  of  control).  Most  of  the  managers  that  we  interviewed,  despite 
their  keen  interest  in  analysis,  are  not  able  to  continue  to  perform  intelligence  assessments 
themselves  due  to  multiple  demands  on  their  time.  However,  in  order  to  monitor  and  evaluate  the 
quality  of  analytic  products  and  processes,  managers  need  to  be  abreast  of  analytic  best 
practices,  current  events,  and  developments  in  areas  being  analysed.  Managers  also  need  to  be 
aware  of  potential  analytic  pitfalls  and  take  steps  to  minimize  their  impact  on  intelligence 
processes  and  products. 

3.7.2  Managerial  challenges 

3.7.2. 1  Performance  evaluation 

Managers  face  a  number  of  challenges  related  to  the  evaluation  of  analytic  products  and  analysts’ 
performance.  One  of  the  difficulties  in  this  regard  is  associated  with  the  absence  of  clear  and 
consistent  feedback  from  intelligence  consumers,  which  frustrates  assessments  of  the  impact  of 
analytic  products  and  whether  they  fulfil  consumers’  needs.  This  issue  was  discussed  in  Sub¬ 
section  3.2.2,  and  in  the  present  section,  we  focus  on  other  challenges  related  to  performance 
evaluation  that  managers  identified: 

•  challenging  analysts, 

•  maintaining  objectivity  in  evaluating  a  product  in  which  the  manager  has  been  closely 
involved,  and 

•  inconsistency  in  performance  evaluation  requirements  across  managers. 

We  briefly  discuss  these  issues  in  the  following  subsections. 

3.7.2. 1.1  Challenging  analysts 

Results.  One  difficulty  in  performance  evaluation  that  some  managers  identified  is  associated 
with  challenging  analysts  on  their  assumptions  and  thinking  processes.  These  managers  indicated 
that  it  is  their  responsibility  to  question  analyst’s  judgments  and  analytic  process  in  order  to 
ensure  the  rigour  and  overall  quality  of  the  product.  As  we  noted  earlier,  it  is  psychologically 
difficult  for  analysts  to  objectively  evaluate  diverging  perspectives  on  the  issue  without  being 
defensive.  In  addition  to  dealing  with  analysts’  defensiveness  in  this  process,  managers  usually 
have  less  information  on  the  issue  than  their  analyst  and  can  only  spend  a  limited  amount  of  time 
thinking  about  and  researching  the  topic.  This  contributes  to  the  uncertainty  that  managers  may 
experience  during  the  challenging  process,  making  it  more  difficult. 

Challenging  analysts  was  very  difficult,  especially  on  issues  that  you  are  not 
familiar  with.  You  feel  that  the  questions  you  are  asking  are  really  benign 
questions  ...  .  There  are  “difficult”  analysts  who  won’t  budge  from  their 
judgments.  With  them,  you  really  have  a  hard  time  being  convinced  that  what 
they  are  arguing  is  the  right  conclusion.  If  you  don’t  have  all  of  the  background 
information  and  you  are  not  convinced,  there  is  a  fine  line  between  being  unsure 
with  what  they  are  bringing  to  you  and  trusting  that  they  are  the  experts.  You 
have  to  have  the  ability  to  be  forceful,  stay  on  top  of  the  issues  so  that  you  know 
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how  and  where  to  challenge  your  analysts.  At  the  same  time,  you  have  to  trust 
your  analysts.  [There  is  a]  Fine  line  between  being  too  hands-on  and  completely 
letting  go.  (Interviewee) 

Some  managers  indicated  that  the  two  requirements  for  effectively  challenging  analysts  are 
comprehensive  knowledge  of  the  subject  area  and  a  good  grasp  of  the  analytic  process.  Some 
managers  also  pointed  out  the  lack  of  training  for  managers  on  how  to  challenge  analysts  and 
suggested  that  it  may  be  helpful  to  develop  a  standard  “challenging  mechanism”  that  could  be 
used  regardless  of  the  topic  or  a  manager’ s  familiarity  with  it. 

Discussion.  Intelligence  managers  cannot  rely  on  standard  mechanisms  for  performance  and 
product  evaluation;  each  report  requires  unique  attention.  In  order  to  be  able  to  challenge 
analysts’  assumptions  and  thought  processes,  a  manager  needs  to  have  a  certain  level  of 
knowledge  about  the  topic  that  is  analyzed.  The  fact  that  managers  usually  supervise  a  group  of 
analysts  who  deal  with  various  (though  related)  issues  increases  the  demand  on  the  manager,  who 
needs  to  remain  informed  on  a  variety  of  topics. 

To  effectively  challenge  their  analysts,  managers  also  need  to  be  competent  analysts  themselves, 
to  be  aware  of  the  potential  pitfalls  in  judgment,  and  to  be  knowledgeable  about  information 
sources.  These  requirements  make  it  more  appealing  to  select  intelligence  managers  from  the 
limited  pool  of  competent  analysts,  which  may  limit  succession  opportunities  available  for 
intelligence  organizations.  In  addition  to  background  knowledge  and  analytic  skill  and 
experience,  the  challenging  process  requires  appropriate  social  skills.  As  some  managers 
indicated,  trust  between  a  manager  and  members  of  his  or  her  analytic  team  plays  an  important 
role  in  the  product  evaluation  process.  Also,  the  manager’s  skillful  approach  to  challenging 
analysts’  thinking  and  assumptions  may  affect  the  degree  of  confrontation  between  them.  As 
some  managers  pointed  out,  there  had  been  no  training  geared  towards  managers  to  prepare  them 
for  these  challenges.  However,  since  our  interviews,  the  IALP  has  designed  and  begun  to  offer  a 
training  course  for  Canadian  intelligence  managers. 

3.7.2. 1.2  Inconsistency  in  performance  evaluation  requirements 

Results.  Some  managers  also  commented  on  the  lack  of  consistency  in  requirements  for  product 
quality  across  various  departments  and  organizations. 

My  view  is  that  the  quality  of  papers  across  the  [organization]  does  vary  quite 
considerably.  It  varies  by  analyst  and  also  by  [section]  ...  there  is  no  qualitative 
check  or  standard,  and  I  think  there  needs  to  be  [ one ].  (Interviewee) 

IAS  managers  commented  that  in  an  attempt  to  standardize  the  evaluation  process,  the  IAS  has 
implemented  quantitative  requirements  for  their  analysts,  according  to  which  analysts  are 
required  to  produce  a  certain  number  of  reports  each  year,  as  determined  by  their  experience  and 
the  position  held.  Interviewed  managers  from  CDI  did  not  foresee  the  implementation  of  similar 
performance  standards  in  their  organization. 

Discussion.  There  is  a  lack  of  consensual  guidelines  to  evaluate  the  quality  of  analytic 
assessments.  Thus,  the  evaluation  process  in  intelligence  analysis  is  largely  subjective  (Johnston 
2005,  Marrin  and  Clemente  2006).  Different  managers,  depending  on  their  own  philosophy, 
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experience,  background  and  pressures,  may  have  differing  notions  and  approaches  to  evaluating 
analytic  products  and  analysts’  performance.  IAS’s  initiative  in  establishing  quantitative 
standards  for  their  analysts  is  a  move  towards  more  objective  performance  evaluation,  as  it 
provides  means  to  measure  performance  based  on  outcomes.  Herman  (1996)  argued  that  seeking 
quantitative  approaches  to  evaluating  intelligence  and  its  impact  should  be  part  of  modern 
intelligence  culture.  Since  the  quality  of  analytic  reports  cannot  be  easily  quantified,  measurement 
focuses  instead  on  the  number  of  reports  an  analyst  is  expected  to  produce  per  annum.  Some 
managers  stressed,  however,  that  the  papers  must  be  of  of  sufficient  quality.  The  burden  is  on 
managers  to  ensure  the  appropriate  level  of  quality  and  complexity  in  each  report,  and  that 
analysts  are  not  just  "pumping  up  their  numbers",  as  one  interviewee  phrased  it.  Interviewed 
managers  from  IAS  welcomed  the  new  quantitative  standards  because  they  provide  an  objective 
performance  evaluation  criterion  across  departments. 

Although  we  did  not  ask  about  analysts’  attitudes  toward  this  initiative,  the  increased  pressure  on 
analysts  to  produce  a  certain  number  of  reports  may  lead  them  to  focus  on  quantity  of  output  at 
the  expense  of  quality.  Such  a  shift  in  emphasis  is  known  as  goal  displacement  resulting  from 
performance  measurement  and  reward  systems.  In  his  classic  article  “On  the  folly  of  rewarding 
A,  while  hoping  for  B”,  Kerr  (1995)  provides  numerous  examples  of  the  goal  displacement 
phenomenon  in  various  organizational  settings.  The  potential  goal  displacement  among  analysts, 
in  turn,  would  increase  the  burden  on  managers,  who  are  left  to  resist  this  displacement  and  to 
ensure  the  satisfactory  quality  of  reports.  In  addition,  ensuring  the  quality  of  reports  will  become 
more  challenging  for  a  manager  as  the  number  of  reporting  analysts  increases. 

3.7.2. 2  Other  challenges 

Results.  In  addition  to  difficulties  associated  with  evaluating  the  performance  of  analysts  and  the 
quality  of  outcomes,  different  managers  commented  on  a  number  of  other  challenges,  which 
included: 

•  Dealing  with  underperformers:  Some  managers  pointed  out  that  they  have  inadequate 
procedures  to  deal  with  underperforming  analysts.  They  would  rather  tolerate  an 
underperforming  analyst  than  get  involved  in  a  time  consuming  and  ineffective  formal 
sanctioning  process. 

•  Learning  on  the  job:  Some  managers  pointed  out  that  there  had  been  no  specialized  training 
for  managers  in  the  intelligence  domain,  and  they  had  to  learn  on  the  job  how  to  deal  with 
specific  issues  that  arise.  It  was  pointed  out  that  some  managers  rely  on  their  former 
superiors’  patterns  of  behaviour  that  they  observed  when  they  were  analysts  themselves. 

•  Span  of  control:  As  the  number  of  reporting  analysts  increases,  it  becomes  more  challenging 
for  managers  to  monitor  the  analysis  process,  to  adequately  challenge  their  analysts,  and  to 
ensure  the  quality  of  reports. 

•  Distributing  limited  budget  resources  for  professional  development  opportunities  for 
analysts  is  not  easy,  as  budgets  frequently  seem  to  allow  for  less  than  what  is 
ideally  desired. 

•  Transitioning  into  a  managerial  role  is  difficult,  not  only  due  to  a  lack  of  management 
training,  but  also  because  it  is  difficult  to  let  go  of  doing  analysis.  Managers’  new 
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responsibilities  do  not  allow  them  to  continue  practicing  analysis  as  much  as  they 
would  like. 

•  Some  managers  also  commented  that  they  sometimes  have  to  deal  with  the  personal 
problems  of  their  analysts,  which  can  be  quite  challenging  at  times. 

Discussion.  Some  of  the  difficulties  that  managers  confront  are  caused  by  organizational  issues 
such  as  human  resource  practices,  which  regulates  procedures  for  dealing  with  underperformers, 
and  organizational  structure,  which  determines  a  manager’s  span  of  control  and  budget  allocation. 
Dealing  with  these  concerns  requires  addressing  wider  organizational  issues.  Yet  some  other 
challenges  may  be  mitigated  through  specially  designed  training  for  intelligence  managers  and 
support  groups,  in  which  managers  could  share  their  concerns  with  other  managers,  learn  from 
each  other’s  experiences,  share  ideas,  and  provide  coaching  to  each  other.  Implementation  of  such 
coaching  circles  for  the  director-level  management  personnel  in  industry  has  had  positive  results. 

3.8  The  Canadian  intelligence  community  in  general 

Results.  Some  of  the  managers  that  we  interviewed  pointed  out  that  there  has  not  been  a  tradition 
of  intelligence  analysis  in  Canada  and  that  intelligence  as  a  profession  is  not  well  defined  in  the 
Canadian  community.  Recently,  there  have  been  organized  efforts  towards  standardization  and 
professionalization  of  the  Canadian  IC  through  such  bodies  as  the  Intelligence  Assessment 
Coordination  Committee  (IACC),  IALP,  and  Canadian  Association  of  Professional  Intelligence 
Analysts  (CAPIA),  whose  mandate  is  “to  make  analysis  a  discipline  instead  of  just  a  practice” 
(interviewee).  Some  managers  also  pointed  out  that  organizing  a  community-wide  introductory 
intelligence  analysis  course  is  a  great  step  toward  analysis  professionalization,  but  much  more 
needs  to  be  done. 

As  some  of  our  interviewees  noted,  Canada’s  IC  is  significantly  smaller  than  the  intelligence 
communities  of  its  allies  (e.g.,  the  US  and  the  United  Kingdom  (UK)).  Having  a  smaller  size,  the 
Canadian  community  is  still  faced  with  a  multitude  of  issues  that  result  in  some  agencies  being 
understaffed,  ultimately  putting  greater  demand  on  individual  analysts.  Some  managers 
commented  that  a  community’s  size  has  an  impact  on  its  methodological  traditions  and  that 
various  analytic  tools  and  techniques  are  not  widely  applied  in  the  Canadian  community. 
According  to  some  managers,  analytic  tools  require  a  time  investment  to  master  and  their 
application  adds  to  the  time  required  to  complete  an  assessment.  Because  of  the  constant  pressure 
to  produce,  analysts  cannot  afford  to  spend  extra  time  to  learn  and  apply  new  analytic  tools  and 
techniques.  As  a  result,  the  Canadian  community  cannot  cultivate  the  expertise  in  these 
methodological  areas  that  is  necessary  for  developing  community  training  programs  and 
providing  support  to  analysts  in  their  application  of  the  techniques. 

The  relatively  small  size  of  the  Canadian  community  also  limits  the  number  of  available  experts 
in  a  given  area,  resulting  in  prospective  intelligence  consumers  shaping  analytic  assessments 
(e.g.,  through  the  IEGs).  Some  managers  that  we  interviewed  see  this  involvement  as  inevitable 
due  to  the  size  of  the  Canadian  community,  and  beneficial  in  bringing  analysts  and  consumers 
closer  together. 

Discussion.  The  Canadian  IC  is  not  unique  in  overlooking  the  role  of  intelligence  analysis. 
Johnston  (2005)  argued  that  the  emphasis  in  the  US  intelligence  community  for  a  long  time  had 
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been  on  collection  and  not  on  analysis;  consequently,  Bruce  and  George  (2008)  commented  on 
the  scarcity  of  literature  devoted  to  issues  of  analysis.  The  intelligence  capability  of  Canada’s 
allies  has  been  subjected  to  scrutiny,  drawing  attention  to  the  pivotal  role  of  analysis  and 
emphasising  the  need  for  significant  changes.  Unlike  its  allies,  the  Canadian  IC  has  not 
undergone  similar  review  and  does  not  have  as  strong  of  an  external  pressure  to  change.  Efforts 
towards  professionalization  of  intelligence  analysis  in  the  Canadian  community  have  been 
internally  driven.  The  IACC  was  stood  up  in  2006.  Its  mandate  includes  performing  assessments 
of  the  IC  and  addressing  issues  that  arise,  such  as  training,  professional  development,  and  a 
common  human  resource  strategy.  The  IALP  was  established  in  September  2005  in  response  to 
demands  raised  by  analysts  for  more  training.  The  IALP  is  acting  on  behalf  of  the  IACC. 
Professional  training  courses  have  been  offered  through  the  IALP  for  analysts  and  managers.  Lor 
example,  a  ten-day  Entry-Level  Course  for  Intelligence  Analysts  and  a  three-day  Managers  of 
Intelligence  Analysts  Course  have  been  offered  on  a  regular  basis  and  are  open  to  analysts  and 
managers  from  various  organizations  in  the  community.  The  IALP  also  offers  a  number  of 
specialized  courses  to  members  of  the  Canadian  intelligence  community.  Lor  instance,  one  of  us 
(David  Mandel)  taught  a  one-day  seminar,  on  judgment  and  decision  making  under  conditions  of 
uncertainty,  as  part  of  the  IALP’s  2008  program  offerings.  In  addition,  CAPIA  organizes  various 
workshops  for  analysts  to  address  issues  in  their  profession.  Information  about  course  and 
workshop  schedules  is  disseminated  through  a  monthly  Intelligence  Analyst  Training  Newsletter, 
published  by  the  IALP,  which,  in  addition  to  training  information,  also  contains  articles 
discussing  various  issues  related  to  intelligence  analysis.  The  present  authors  have  contributed  to 
this  newsletter  (Derbentseva  and  Mandel  2010b,  Mandel  2010). 

However,  different  organizations  within  the  Canadian  community  have  their  own  understanding 
of  the  core  standards  and  competencies  for  intelligence  analysts,  perhaps  derived  from  their 
unique  functions.  Lack  of  agreement  on  essential  capabilities  for  analysts  significantly  hinders  the 
development  of  a  community-wide  training  program  that  would  address  the  needs  of  the 
community  as  a  whole.  Lurthermore,  some  managers  expressed  concern  that  it  would  be  difficult 
to  reach  consensus  on  these  issues  among  departments. 

Compared  to  its  allies,  the  Canadian  community  has  a  relatively  small  size,  which  offers  both 
potential  opportunities  and  drawbacks.  Lor  instance,  as  a  smaller  community,  the  Canadian  IC  has 
constraints  for  employing  analysts  as  well  as  for  developing  and  maintaining  support  activities 
required  for  conducting  analysis  and  professional  development  programs.  Some  managers 
attributed  this  lack  of  application  of  structured  analytic  techniques  to  the  lack  of  resources.  There 
are  currently  no  educational  programs  that  prepare  professional  intelligence  analysts,  which  could 
be  attributed  in  part  to  the  lack  of  a  steady  demand  for  such  professionals  in  the  Canadian  IC  . 
Consequently,  there  are  also  no  standard  educational  requirements  for  entering  the  profession. 
Analysts  receive  most  of  their  training  on-the-job  through  mentoring  and  ad-hoc  courses, 
with  the  exception  of  military  intelligence  analysts  who  undergo  a  more  structured  and  systematic 
training  program. 

Also,  the  relatively  small  size  of  the  Canadian  IC  may  offer  certain  advantages.  Lor  example,  it 
may  be  easier  for  analysts  to  get  to  know  their  colleagues  from  other  organizations  working  on 
similar  issues.  Lor  instance,  IEGs  may  provide  a  venue  for  collaboration.  Notably,  the 
community,  with  a  relatively  small  resource  base,  has  put  significant  effort  into  developing 
community-wide  training  for  analysts,  which  has  managed  to  reach  far,  particularly  because  of 
the  relatively  small  community  size.  Additionally,  as  some  managers  pointed  out,  a  smaller 
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community  brings  analysts  and  consumers  closer  together.  Previously,  such  closeness  was  seen  to 
introduce  bias  into  assessments  and  decisions  (e.g.,  Armstrong  1989).  However,  the  current 
prevailing  view  in  the  literature  on  this  issue  argues  for  bridging  the  gap  between  policy  and 
intelligence  analysis  to  ensure  relevance  of  intelligence  to  policy  (Hedley  2007,  McLaughlin 
2008,  Steinberg  2008,  Treverton  2008).  Nevertheless,  the  problem  of  obtaining  feedback  from 
consumers  remains  pertinent. 
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4  Intelligence  analysis:  directions  for  future  research 


One  of  the  main  goals  of  the  present  study  was  to  identify  research  areas  that  could  further  our 
understanding  of  the  intelligence  analysis  process  and  potentially  provide  opportunities  for 
augmenting  it.  The  focus  of  our  discussion  in  this  report  has  been  on  behavioural  aspects  (i.e., 
cognitive,  motivational,  social,  and  organizational)  of  intelligence  analysis.  Since  intelligence 
analysis  is  mainly  a  product  of  human  reasoning  and  social  interactions,  behavioural  sciences, 
broadly  defined  as  various  scientific  disciplines  studying  human  behaviour,  can  play  an  important 
role  in  advancing  our  understanding  of  and  supporting  analytic  activities  (Mandel  2009c,  National 
Research  Council  2010).  The  accumulated  knowledge  and  methodological  approaches  of  the 
behavioural  sciences  could  be  leveraged  to  improve  our  understanding  of  various  issues  in 
intelligence  analysis  identified  in  this  report,  such  as  the  mechanisms  and  impact  of  various 
cognitive  processes  involved  in  information  processing,  reasoning,  and  judgment;  the  role  of 
motivational  factors,  individual  differences,  and  organizational  dynamics  in  the  process  and 
outcomes  of  intelligence  analysis.  Through  better  understanding  of  various  behavioural  factors 
affecting  intelligence  activity,  behavioural  sciences  could  be  applied  to  improving  performance 
on  various  intelligence-related  tasks.  In  order  to  achieve  this  leveraging  successfully,  the 
knowledge  of  both  behavioural  sciences  and  intelligence  analysis  is  required.  An  example  of  the 
integration  and  a  daptation  of  scientific  knowledge  to  developing  practical 
recommendations  for  the  IC  is  Heuer’s  ACH  tool  designed  to  mitigate  the  impact  of  confirmation 
bias  in  hypotheses  evaluation. 

Although  the  integration  of  scientific  findings  can  and  has  been  undertaken  from  within  the  IC 
(e.g.,  Heuer  1999),  Mandel  (2009c)  argued  that  the  IC’s  reliance  on  lone  “mavericks”  like  Heuer 
is  not  an  effective  or  proactive  approach  for  the  community  to  stay  abreast  of  scientific 
developments,  and  that  there  are  a  number  of  reasons  for  the  IC  to  outreach  to  a  broader  scientific 
community  in  fulfilling  its  R&D  requirements.  Mandel  (2009c,  National  Research  Council  2010) 
proposed  that  developing  a  partnership  between  the  IC  and  the  behavioural  sciences  community 
can  facilitate  the  IC’s  timely  exploitation  of  the  knowledge  accumulated  in  various  pockets  of  the 
scientific  community  without  diverting  the  IC’s  resources  from  its  primary  mandate  of  producing 
intelligence.  The  practical  value  of  the  partnership  between  the  intelligence  and  behavioural 
sciences  communities  was  also  recognized  at  the  2009  Ottawa  roundtable  of  the  Global 
Futures  Forum’ s  (GFF)  Community  of  Interest  on  the  Practice  and  Organization  of  Intelligence 
(COI  POI)  that  brought  together  intelligence  professionals  and  the  scientific  community  to 
discuss  the  potential  contribution  of  behavioural  sciences  to  the  area  of  intelligence  (Campbell 
and  Mandel  2010). 

The  foregoing  examination  of  managers’  responses  to  our  interview  questions  suggests  several 
areas  for  extending  R&D  efforts,  in  which  the  behavioural  sciences  could  be  utilised  to  improve 
our  understanding  of  various  cognitive  and  behavioural  phenomena  pertinent  to  the  intelligence 
analysis  domain.  These  areas  are  outlined  in  Figure  6  and  are  discussed  briefly  in  the  following 
Sections.  These  research  areas  can  be  mapped  onto  the  four  dimensions  of  people,  processes, 
technology,  and  organization  of  the  intelligence  analysis  capability  development.  For  example, 
“selection  process,”  “performance  evaluation,”  and  “training”  research  areas  from  Figure  6 
address  the  “People”  dimension  of  capability  development.  Similarly,  “process  of  analysis,” 
“consumer-producer  relationship,”  “performance  evaluation,”  “knowledge  management,” 
“training,”  and  “information  management”  research  areas  from  Figure  6  target  the  “Process” 
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aspect  of  capability  development.  Both  “information  management”  and  “knowledge 
management”  research  areas  address  the  “Technology”  aspect  of  intelligence  analysis  capability 
development.  Lastly,  “consumer-producer  relationship”  research  area  pertains  to  the 
“Organization”  dimension  of  capability  development. 


Figure  6:  Areas  for  further  investigation 

4.1  The  process  of  intelligence  analysis 

As  we  commented  earlier,  until  fairly  recently,  research  on  analysis  had  not  received  a  great  deal 
of  attention  in  the  international  intelligence  community.  Consequently,  the  area  of  intelligence 
analysis  does  not  have  an  abundant  body  of  literature,  processes  or  best  practices  that  are  as  well 
defined  as  other  disciplines,  e.g.,  medicine  or  law  (Bruce  and  George  2008,  Johnston  2005, 
Marrin  and  Clemente  2005).  Research  on  analysis  within  the  Canadian  intelligence  community  is 
unsurprisingly  much  scarcer. 

We  propose  that  more  effort  ought  to  be  put  into  understanding  how  intelligence  analysis  is 
actually  done,  investigating  underlying  behavioural,  cognitive,  and  social  processes  in 
intelligence  analysis,  and  systematically  evaluating  tools  and  techniques  that  are  currently 
employed  by  analysts  or  being  considered  for  adoption.  In  the  sub-sections  that  follow,  we  focus 
on  each  of  these  issues  in  more  detail. 
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4.1.1  Investigation  of  how  analysis  is  actually  done 


Most  managers  commented  that  currently  the  “intuitive”  approach  to  analysis  is  predominant  in 
the  community,  whereby  analysts  read  a  lot,  think,  and  write  a  report.  However,  the  underlying 
processes  of  how  analysts  search  and  select  what  to  read;  how  they  evaluate,  interpret  and  use 
what  they  read;  and  what  affects  and  directs  their  thinking  processes  and  so  on  are  not  well 
described  or  understood.  Understanding  the  actual  process  of  analysis  requires  a  deeper 
investigation  and  better  description.  More  insight  into  the  processes  involved  in  analysis  can  be 
achieved  through  cognitive  task  analysis,  interviews  with  intelligence  analysts,  observation  of 
their  activities,  and  socio-technical  network  analyses  of  the  intelligence  analyst  role.  Application 
of  these  methods  will  reveal  cognitive  processes  and  workload,  their  contingencies,  flows  of 
information,  and  bottlenecks  in  the  analysis  process.  Obtaining  a  comprehensive  description  of 
the  analysis  process  can  help  to  uncover  additional  cognitive,  motivational,  or  organizational 
issues,  that  will  assist  in  identifying  training  and  tool  development  needs,  and  help  to  further  the 
professionalization  of  intelligence  analysis. 

4.1.2  Cognitive  processes  in  analysis 

Intelligence  analysis  is  predominantly  a  cognitive  activity  that  relies  heavily  on  human  judgment 
(e.g.,  Heuer  1999,  Moore  2007).  Understanding  the  analytic  process  greatly  depends  on 
identifying  and  understanding  the  underlying  cognitive  processes  involved.  Heuer  in  his  seminal 
work  “Psychology  of  Intelligence  Analysis”  reinterpreted  much  of  the  knowledge  accumulated  in 
cognitive  sciences  into  terms  that  are  relevant  to  intelligence  analysis.  Although  Heuer’ s  book 
was  published  in  1999,  it  represents  a  collection  of  articles  that  he  wrote  from  1978  -  1986  for 
internal  use  within  the  CIA  (author's  preface,  Heuer  1999).  Consequently,  most  of  the  research 
that  Heuer  reviewed  in  his  book  was  published  in  the  1970s  and  1980s  (Mandel  2009c).  Heuer’ s 
work  remains  a  valuable  sourcebook  that  integrates  psychological  issues  into  the  domain  of 
intelligence  analysis.  However,  a  decade  has  passed  since  the  publication  of  Heuer’ s  book  and 
roughly  a  quarter  century  of  new  behavioural  science  research  has  been  conducted  since  Heuer’ s 
review.  In  the  intervening  period,  there  has  been  no  systematic  effort  to  incorporate  this  new  body 
of  work  (Mandel  2009c).  To  be  accessible  and  useful  to  the  intelligence  community,  these 
findings  need  to  be  interpreted  in  intelligence -relevant  terms  keeping  the  community’s  needs  in 
mind.  With  support  from  intelligence  experts,  the  behavioural  science  community  could  take  the 
lead  in  updating  “Psychology  of  Intelligence  Analysis”  with  more  recent  scientific  findings,  as 
well  as  providing  continuous  support  in  maintaining  this  resource  up  to  date.  This  effort  will 
ensure  that  the  most  recent  developments  in  the  behavioural  sciences  are  at  the  intelligence 
community’s  disposal,  ready  to  be  exploited. 

In  addition  to  utilising  findings  accumulated  by  the  behavioural  sciences,  close  examination  of 
the  analysis  process  may  reveal  additional  cognitive  and  behavioural  issues,  which  may  be  unique 
to  the  process  of  analysis.  During  our  interviews,  managers  commented  on  a  number  of  cognitive 
processes  involved  in  intelligence  analysis.  Some  of  these  issues  require  further  investigation 
including  the  impact  of  information  secrecy  on  perceptions  of  information  accuracy  and  value, 
the  impact  of  information  quality  on  the  resulting  judgment,  analysts’  decision  avoidance,  the 
relationship  between  expertise  and  confirmation  bias,  and  the  impact  of  time  pressure  on 
cognitive  processes  involved  in  analysis.  We  discuss  these  issues  in  more  detail  below. 
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4.1 .2.1  Impact  of  information  secrecy 


One  of  the  issues  mentioned  by  some  managers  was  the  tendency  among  analysts  and  intelligence 
consumers  to  pay  greater  attention  to  classified  information  as  opposed  to  open  source 
information.  Classified  information  creates  an  impression  of  secrecy  and  importance,  and  it  is 
believed  to  have  more  face  validity  than  information  available  through  open  sources  (Johnston 

2005) .  McLellan  et  al.  (2008)  refer  to  this  purported  tendency  to  overinflate  the  value  of  exclusive 
(i.e.,  classified)  information  as  the  secrecy  bias.  However,  the  secrecy  bias  might  not  be  unique  to 
the  intelligence  profession.  Rather,  we  hypothesize  that  it  reflects  a  more  widespread  belief  in  the 
plausible  supposition  that  if  information  is  held  in  secrecy,  then  it  must  have  the  potential  to 
influence,  and,  thus  it  must  have  probable  value.  Although  the  presence  of  this  phenomenon  has 
been  recognized  in  both  the  US  and  Canadian  intelligence  communities  (Johnston  2005, 
Lieberthal  2009),  more  research  is  needed  to  better  understand  its  nature  and  impact. 

4.1 .2.2  Impact  of  information  quality  on  judgment 

Most  of  the  managers  commented  that  intelligence  analysts  have  to  gauge  the  quality  of 
information  they  use  in  their  assessments  and  take  the  quality  of  the  information  into  account  in 
their  judgments.  According  to  several  managers,  information  deliberately  collected  (e.g.,  from 
HUMINT  or  other  intelligence  sources),  which  may  be  available  to  analysts,  often  comes  with  an 
assessment  of  quality,  such  as  source  reliability  and  information  credibility.  We  have  limited 
understanding  of  how  analysts  gauge  quality  of  information  and  how  they  incorporate  these 
quality  assessments  into  their  judgments.  This  is  a  particularly  challenging  task  in 
situations  with  multiple  pieces  of  information  of  varying  quality  and  diagnosticity.  A  better 
understanding  of  how  analysts  incorporate  the  quality  of  information  into  their  judgments  is 
important  for  developing  sound  analytic  practices.  Preliminary  studies  along  this  line  are 
currently  being  conducted  at  DRDC  Toronto  (Tombu  and  Mandel  2010),  and  more  research  is 
needed  in  this  area. 

4.1 .2.3  Mental  representations  and  estimative  judgments 

One  factor  that  may  affect  judgments  about  a  possible  future  event  is  one's  mental  representation 
of  that  event.  Construal  Level  Theory  (CLT;  Trope  and  Liberman  2003)  posits  the  argument  that 
individuals  use  low-level  construals  to  represent  events  that  are  psychologically  near,  and  use 
high-level  construals  to  represent  events  that  are  psychologically  distant.  Low-level  construals  are 
contextualized  representations  that  include  concrete  features  of  the  event  whereas  high-level 
construals  are  schematic  representations  that  include  abstract  features  of  the  event.  For  example, 
helping  to  build  a  school  in  Afghanistan  can  be  construed  on  a  low  level  (e.g.,  laying  bricks  and 
mortar)  or  on  a  high  level  (e.g.,  providing  humanitarian  aid).  Mandel  and  McLellan  of  DRDC 
Toronto  are  conducting  experimental  research  based  on  previous  findings  (see  Wakslak  et  al. 

2006)  to  test  the  hypothesis  that  focusing  on  another's  capability  (i.e.,  a  low-level  construal)  will 
cause  people  to  make  higher  probability  estimates  and  shorter  time  estimates  than  will  focusing 
on  another’s  motive  (i.e.,  a  high-level  construal;  McLellan  et  al.  2010a,  McLellan  and  Mandel 
2010a).  Event  construal  is  manipulated  by  having  participants  focus  on  how  an  event  might  occur 
(to  induce  low-level  construal)  or  why  an  event  might  occur  (to  induce  high  level  construal). 
Questions  about  an  adversary’s  capabilities  and  motives  are  commonly  addressed  in  the 
intelligence  process.  For  example,  an  analyst  might  consider  why  the  adversary  would  deploy  a 


DRDC  Toronto  TR  2010-144 


69 


dirty  bomb  (i.e.,  what  are  the  motives  underlying  this  action?)  or  how  the  adversary  might  do  so 
(i.e.,  what  are  the  capabilities  required  to  carry  out  this  action?).  CLT  would  predict  that  focusing 
on  how  an  event  might  occur  will  lead  to  higher  probability  estimates  than  focusing  on  why  that 
same  event  might  occur.  If  supported  empirically,  this  effect  could  have  implications  for 
identifying  biases  in  intelligence  analysis  when  making  estimative  judgments  that  focus  on  others' 
capabilities  or  motives. 

4.1 .2.4  Decision  avoidance 

Given  the  high  degrees  of  uncertainty  associated  with  intelligence  analysis,  some  managers 
commented  that  the  difficulty  analysts  experience  with  making  decisions  may  become  disabling 
and  cause  decision  avoidance.  Analysts  may  refrain  from  making  judgments  in  their  assessments 
either  by  not  stating  their  opinion  altogether  or  stating  their  opinion  in  a  vague  form,  allowing  for 
various  interpretations.  Such  assessments  may  be  of  little  value  to  intelligence  consumers 
(Steinberg  2008).  The  two  main  sources  of  decision  avoidance  among  intelligence  analysts 
outlined  in  the  literature  and  mentioned  by  our  interviewees  are  fear  of  being  wrong  (i.e., 
providing  judgments  which  may  later  turn  out  to  be  mistaken)  and  pressure  to  conform  to  the 
predominant  view  among  experts  on  the  issue.  Both  of  these  causes  of  decision  avoidance  most 
likely  have  individual/personality  and  social/situational  antecedents.  A  review  of  the 
psychological  research  conducted  on  this  issue  and  further  investigation  of  its  causes  in  the 
intelligence  analysis  context  may  be  informative  to  both  analysts  and  managers  in  understanding 
the  root  causes  of  this  phenomenon  and  finding  mitigating  factors. 

4.1 .2.5  Confirmation  bias  in  intelligence  analysis 

In  the  intelligence  literature,  confirmation  bias  is  a  frequently  cited  cognitive  propensity  that  is 
believed  to  be  particularly  damaging  to  intelligence  analysis  (e.g.,  Butterfield  1993,  Davis  1992, 
2008,  Heuer  1999).  Some  of  the  most  notorious  intelligence  failures  have  been  attributed  to  the 
perils  of  confirmation  bias  (e.g.,  Yom  Kippur  War,  see  Davis  2008).  The  IC  (predominantly  in 
the  US)  has  put  efforts  into  combating  confirmation  bias  through  the  development  and 
implementation  of  tools  such  as  ACH  (Heuer  1999).  The  IC  borrowed  the  term  confirmation  bias 
from  psychology,  and  in  psychology  a  number  of  different  phenomena  have  been  studied  under 
the  label  of  confirmation  bias  since  the  classic  “rule  discovery”  research  (Klayman  and  Ha  1987, 
Klayman  1995,  McKenzie  2006,  Nickerson  1998,  Wason  1960).  Confirmation  bias  can  refer  to 
the  tendency  to  search  for  evidence  that  confirms  the  currently  held  hypothesis;  the  tendency  to 
test  instances  that  are  consistent  with  the  currently  held  hypothesis,  but  which  can  produce  both 
confirming  and  disconfirming  evidence;  the  tendency  to  give  greater  weight  to  the  confirming 
evidence  while  underweighting  disconfirming  evidence;  and  the  tendency  to  interpret  ambiguous 
information  as  evidence  supporting  the  hypothesis.  Different  interpretations  of  confirmation  bias 
represent  distinct  phenomena,  which  occur  under  different  sets  of  circumstances  and  require 
different  intervention  measures.  Klayman  (1995)  differentiated  between  two  general  meanings  of 
confirmation  bias  as:  a)  “looking  for  the  presence  of  what  you  expect,  as  opposed  to  looking  for 
what  you  do  not  expect,”  which  Klayman  and  Ha  (1987)  termed  positive  test  strategy,  and  b)  “an 
inclination  to  retain,  or  a  disinclination  to  abandon,  a  currently  favoured  hypothesis”  (p.386). 
Klayman  and  Ha  (1987)  argued  that  a  positive  test  strategy,  which  is  often  confused  with 
confirmation  bias,  is  an  adaptive  coping  mechanism  that  can  be  quite  efficient  in  many 
hypotheses  testing  situations  (also  see  Mandel  2010). 
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Various  incidents  in  intelligence  analysis  that  are  labelled  as  instances  of  confirmation  bias 
require  a  closer  examination  in  order  to  differentiate  among  distinct  cognitive  phenomena.  In 
addition,  more  research  is  needed  to  identify  the  conditions  under  which  different  forms  of 
confirmation  bias  may  be  a  concern  in  intelligence  analysis.  For  example,  McKenzie  (2006) 
demonstrated  that  confirmation  bias  occurs  only  under  a  very  limited  set  of  conditions  when 
people  work  on  familiar  tasks.  The  extent  to  which  confirmation  bias  and  overconfidence,  which 
may  be  associated  with  it  (McKenzie  2006),  is  an  issue  in  intelligence  analysis  is  actually  unclear. 
On  the  one  hand,  there  is  a  plethora  of  anecdotal  evidence  of  confirmation  bias’  negative  impact 
reported  in  the  intelligence  literature  (e.g.,  Davis  2008).  On  the  other  hand,  scientific  studies  of 
analytic  judgments  (although  very  scarce)  suggest  otherwise.  For  example,  an  analysis  of  actual 
intelligence  judgments  of  Canadian  analysts  provides  evidence  of  underconfidence  in  judgments, 
which  is  associated  with  the  absence  of  confirmation  bias  (Mandel  2009b).  Similarly,  studies  with 
intelligence  analysis  trainees  on  unfamiliar  tasks  (i.e.,  posterior  probability  assessments)  reported 
by  Mandel  (2009a)  suggest  underconfidence  (and  hence,  the  absence  of  confirmation  bias)  in 
analysts’  judgements,  which  is  reduced  with  training.  In  the  absence  of  confirmation  bias, 
measures  that  are  designed  to  mitigate  it  may  actually  be  detrimental,  because  they  may 
reduce  analysts’  confidence  (i.e.,  increase  their  underconfidence)  and,  thus,  decrease  analysts’ 
judgment  accuracy. 

It  is  also  important  to  investigate  the  relationship  between  expertise  and  susceptibility  to 
confirmation  bias.  Accounts  of  the  hindering  impact  of  expertise  on  recognising  unusual 
developments  have  been  reported  by  our  interviewees  and  in  the  intelligence  literature  (Davis 
2008).  The  relationship  between  expertise  and  confirmation  bias  has  not  received  a  great  deal  of 
attention  in  the  psychological  literature.  An  argument  for  both  mitigating  (e.g.,  see  Klayman 
1995  for  a  review)  and  hindering  (e.g.,  Bilalic  et  al.  2008)  effects  of  expertise  on  the  performance 
of  different  tasks  and  confirmation  bias  have  been  made,  and  fragmented  studies  on  this  topic 
report  contradicting  results.  The  relationship  between  expertise  and  confirmation  bias  is 
especially  pertinent  to  those  intelligence  organizations  that  hone  analysts’  expertise.  Further 
investigation  of  this  relationship  will  contribute  to  a  better  understanding  of  intelligence 
analysis  and  also  will  contribute  to  the  general  bodies  of  knowledge  on  expertise  and 
cognitive  psychology. 

4.1 .2.6  Effect  of  time  pressure  on  thinking  and  reasoning 

Intelligence  assessments  are  usually  time  sensitive,  and  time  pressure  has  been  identified  as  one 
of  the  enduring  characteristics  of  intelligence  analysis  (Johnston  2005,  Treverton  2008).  As  we 
discussed  above  (see  Sub-section  3.2.2.3),  research  in  cognitive  psychology  has  shown  that  time 
pressure  has  an  impact  on  cognitive  processes,  often  in  a  hindering  fashion  (for  a  review,  see 
Maule  and  Edland  1997).  However,  specific  effects  of  time  pressure  on  intelligence  analysis 
processes  and  the  magnitude  of  its  impact  on  the  quality  of  analytic  assessments  may  benefit  from 
additional  directed  attention.  Identification  and  description  of  the  actual  intelligence  analysis 
process  may  be  instrumental  in  designing  a  research  program  designed  to  investigate  the  specific 
impact  of  time  pressure  on  the  cognitive  processes  involved  in  intelligence  analysis  as  well  as 
potential  adaptation  mechanisms  and  mitigating  factors. 
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4.1.3  Methods  for  representing  and  organizing  information  to  promote 
analytic  rigour 

Identifying  appropriate  and  advantageous  ways  to  represent  and  organize  information  for  analysts 
during  their  thinking  process  was  identified  at  a  recent  GFF  workshop  as  an  important  direction 
for  future  behavioural  science  research  in  support  of  intelligence  analysis  (Campbell  and  Mandel 
2010).  Winston  (1992)  pointed  out  that  “once  a  problem  is  described  using  an  appropriate 
representation,  the  problem  is  almost  solved.”  Various  tools  provide  different  ways  of  organizing 
information.  For  example,  using  ACH  (Heuer  1999),  an  analyst  constructs  a  matrix  with  pieces  of 
evidence  listed  down  the  side  and  alternative  hypotheses  listed  across  the  top  (see  Figure  7). 
Such  a  matrix  generates  an  intersection  between  each  piece  of  evidence  and  each  of  the 
hypotheses,  drawing  analysts’  attention  to  the  relationship  between  evidence  and  hypotheses. 


Hypothesis/  Hypothesis/  Hypothesis/  ...  Hypothesis„ 

Evidence/ 

Evidence2 

Evidence/ 


Evidence,„ 


Figure  7:  ACFl  matrix 


However,  evidence  as  it  is  presented  in  ACH  forms  a  list  of  seemingly  independent  statements, 
which  may  not  always  be  the  case,  and  ACH  does  not  provide  the  means  to  account  for  potential 
contingencies.  Interdependencies  and  contingencies  among  variables  may  be  better  captured  in  a 
graph-like  representational  form  such  as  Concept  Maps  (CMaps).  CMaps  are  graphs  of 
interconnected  concepts  with  labelled  relationships,  which  are  intended  to  capture  and  map  the 
author’s  conceptual  understanding  of  a  topic.  CMaps  allow  for  the  creation  of  intricate  knowledge 
models,  which  facilitate  the  preservation  and  sharing  of  ideas.  For  example,  Derbentseva  and 
Mandel  (forthcoming-b)  developed  a  CMap  knowledge  model  of  intelligence  analysis  that  was 
created  to  capture  and  represent  conceptual  interdependencies  in  various  topics  pertinent  to 
intelligence  analysis.  Although  CMapping  has  been  adopted  by  some  intelligence  organizations  in 
the  US  (Hoffman  2008),  it  has  remained  a  relatively  unfamiliar  tool  within  the  Canadian  IC. 
Derbentseva  and  Mandel  organized  a  CMapping  workshop  for  Canadian  intelligence  subject 
matter  experts,  in  which  they  introduced  their  knowledge  model  of  intelligence  analysis  and 
solicited  feedback  on  applications  of  CMapping  within  the  community  (Gauthier  2010).  The 
workshop  participants  identified  a  number  of  potential  uses  for  CMapping  within  the  community, 
such  as  to  structure  analysis,  communicate  among  analysts  and  with  consumers,  foster 
collaboration,  and  expedite  the  learning  of  new  desk  analysts  (Derbentseva  and  Mandel 
forthcoming-a).  Further  work  on  modifying  the  knowledge  model  to  better  suit  the  community’s 
needs,  investigating  other  applications  of  the  CMapping  tool,  and  supporting  the  community’s 
CMapping  efforts  are  currently  underway  at  DRDC  Toronto.  For  instance,  a  short  description  of 
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CMapping  as  a  structured  analytic  technique  appeared  in  the  March  2010  issue  of  the  Intelligence 
Analysis  Training  Newsletter  (Derbentseva  and  Mandel  2010b),  and  a  CMap  representation  of 
analytic  rigour  and  standards  is  now  included  in  the  Aide  Memoire  on  Intelligence  Analysis 
Tradecraft  (Thompson  2010).  Analysts  may  benefit  from  tools  that  not  only  allow  them  to  search 
for  information  more  effectively,  but  those  that  also  facilitate  thinking  and  discovery.  Both 
CMapping  and  ACH  seem  to  fit  these  requirements,  however,  more  research  is  required  to 
comprehensively  evaluate  the  potential  benefits  and  shortcomings  of  the  various  available 
representation  systems. 

4.1.4  Analytical  tools  and  techniques 

In  order  to  mitigate  cognitive  difficulties  related  to  information  overload  and  cognitive  biases,  the 
intelligence  communities  of  Canada’s  allies  have  been  designing  and  developing  various  tools 
that  may  help  analysts.  As  a  result,  a  variety  of  tools  has  been  developed  and  applied  in 
intelligence  analysis.  Some  managers  that  we  interviewed  pointed  out  that  tools  are  not  used  as 
much  in  the  Canadian  community  because  of  the  limited  resources  available  for  learning  and 
applying  them.  In  addition,  some  managers  commented  that  there  may  be  a  certain  degree  of 
resistance  to  tools  among  some  analysts  because  “ there  is  no  magic  tool  that  you  can  use  to  get 
the  right  answer.”  Such  attitudes  may  be  the  result  of  past  experience  with  tools  where  they  have 
been  found  to  be  inadequate.  This  attitude  may  also  reflect  on  the  absence  of  systematic 
evaluation  and  the  lack  of  evidence  that  tool  usage  “provides  better  results  than  unaided  expert 
judgment”  (Heuer  and  Pherson  2010,  p.309).  Scientific  evaluation  of  tools  may  shed  more  light 
on  the  applicability,  advantages,  and  limitations  of  various  tools  and  techniques  to  support 
analysis.  Although  there  is  value  in  conducting  tool-evaluation  studies  with  naive  subjects 
(i.e.,  non-analysts)  and  generic  situations  in  order  to  establish  “proof  of  concept,”  Heuer  and 
Pherson  (2010)  stressed  the  importance  of  ecological  validity  of  situations  and  tasks  used  in  the 
evaluation  of  structured  analytic  techniques  for  intelligence  analysis.  Thus,  studies  that  intend  to 
shed  light  on  the  applicability  and  utility  of  analytic  tools  for  intelligence  analysis  need  to  be 
conducted  with  intelligence  analysts  and  under  test  conditions  that  are  as  similar  as  possible  to  the 
analysts’  actual  working  conditions. 

4.1 .4.1  Identifying  the  most  prevalent  tools  used  in  the  Canadian  community 

Some  of  the  managers  commented  that  analysts  are  aware  of  a  variety  of  analytic  tools;  however, 
the  tools  are  not  used  extensively  in  the  Canadian  community.  Given  the  abundance  of  existing 
analytic  tools  and  their  infrequent  application,  it  will  be  beneficial  to  identify  which  tools  are 
available  to  Canadian  analysts,  which  tools  analysts  actually  apply,  and  which  ones  do  they  find 
useful  in  their  work.  Conducting  such  a  survey  will  be  instrumental  in  directing  the  development 
of  a  tool  evaluation  program. 

4.1 .4.2  Evaluating  analytic  tools 

Analytic  tools  are  developed  to  make  the  task  of  intelligence  analysis  easier  and  to  reduce  the 
effect  of  cognitive  biases  inherent  in  human  reasoning.  A  considerable  number  of  analytic 
techniques  have  been  developed  and  introduced  into  the  intelligence  community  (CIA  2005, 
2009,  Davis  1992,  DIA  2008,  Heuer  1999,  Jones  1998).  Many  of  the  techniques,  however,  have 
not  been  systematically  evaluated,  and  the  literature  on  analytic  technique  evaluation  is  relatively 
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sparse  and  fragmented  (Heuer  and  Pherson  2010,  Johnston  2005).  Examples  of  a  small  number  of 
quantitative  studies  aimed  at  investigating  the  effect  of  analytic  techniques  include  work  by 
Folker  (2000),  Cheikes  et  al.  (2004)  and  Pirolli  (2006)  that  evaluated  ACH.  Folker  (2000)  tested 
the  effectiveness  of  ACH  compared  to  an  intuitive  approach  and  found  that  it  facilitated  arriving 
at  the  correct  solution  in  only  one  of  two  scenarios  tested.  Cheikes  et  al.  (2004)  investigated  the 
effectiveness  of  ACH  in  mitigating  confirmation  bias  and  also  found  mixed  results.  ACH 
mitigated  confirmation  bias  only  in  participants  with  no  prior  analysis  experience.  Pirolli  (2006) 
evaluated  the  application  of  ACH  under  different  conditions  but  did  not  include  a  control  group 
that  did  not  use  ACH.  In  the  first  study,  Pirolli  compared  a  paper  ACH  method  with  a 
computerised  ACH  tool  and  did  not  find  significant  differences  in  how  participants  structured  the 
problem  or  on  performance.  In  the  second  study,  Pirolli  examined  the  utility  of  ACH  in  mitigating 
confirmation  bias  in  groups  and  found  mixed  results.  The  initial  bias  decreased  in  individuals 
working  alone  and  in  heterogeneous  groups,  but  the  initial  bias  increased  in  homogeneous  groups. 
Moreover,  the  observed  results  cannot  be  conclusively  attributed  to  the  application  of  ACH  as  the 
study  did  not  include  a  control  group  that  did  not  use  the  tool.  Consequently,  the  results  of  various 
studies  evaluating  ACH  while  somewhat  promising,  are  inconclusive  and  call  for  further 
investigation.  Moreover,  Heuer  and  Pherson  (2010)  questioned  the  external  and  construct  validity 
of  some  of  the  empirical  tests  of  ACH. 

As  we  argued  above,  there  is  a  need  for  a  systematic  evaluation  of  the  tools  and  techniques  used 
in  the  intelligence  domain.  Developing  a  research  program  that  will  rigorously  evaluate  analytic 
techniques  might  be  of  value  to  the  intelligence  community  as  it  will  allow  for  more  educated 
choices  about  which  tools  to  employ.  A  systematic  research  program  will  not  only  investigate 
whether  or  not  a  given  technique  does  indeed  reduce  a  certain  bias,  but  it  will  also  examine  what 
aspects  of  the  technique  make  it  effective.  It  will  examine  the  application  of  techniques  under 
different  conditions  and  varying  task  structures,  and  will  provide  an  overall  better  understanding 
of  the  tool.  For  example,  ACH  (Heuer  1999)  combines  several  analytic  sub-methods,  such  as: 

•  externalising  evidence,  hypotheses,  and  assumptions; 

•  generating  an  (exhaustive)  set  of  hypotheses  before  evaluating  them  against  the  evidence; 
and 

•  creating  lists  of  evidence  and  hypotheses; 

•  systematically  evaluating  each  piece  of  evidence  with  respect  to  each  hypothesis; 

•  paying  attention  to  disconfirming  (rather  than  confirming)  evidence. 

One  possible  direction  for  future  research  would  be  to  examine  the  relative  contribution  of  the 
methods  employed  in  ACH  under  different  conditions.  Another  research  direction  could 
challenge  the  underlying  assumptions  of  ACH  and  examine  the  method’s  robustness.  Other  tools 
may  be  evaluated  following  this  approach.  As  Heuer  and  Pherson  (2010)  suggested,  evaluation  of 
tools  should  employ  a  variety  of  methods  such  as  surveys  of  analysts  who  use  the  tools, 
observations  of  analysts  at  work,  and  structured  interviews  with  analysts,  managers,  and 
consumers.  In  addition,  tool  evaluation  ought  to  rely  on  a  variety  of  dependent  measures  (i.e., 
measures  of  quality)  such  as  accuracy,  clarity  of  presentation,  transparency  of  the  analytic  process 
and  conclusions,  and  construction  of  a  record  of  the  analytic  process. 
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4.2  Information  management 


Intelligence  analysis  is  a  highly  information-dependent  activity.  Both  psychological  issues  related 
to  information  processing  and  systemic  issues  related  to  information  flows  within  and  between 
organizations  have  a  significant  impact  on  intelligence  production. 

4.2.1  Study  of  systemic  issues:  Identifying  breakdowns  between 
information  requirements  and  existing  flows 

The  quality  of  analysis  depends  on  the  availability  of  relevant  information,  and  the  quality  of  the 
information  on  which  analysts  rely  in  making  their  assessments.  The  ability  to  find  relevant 
information  and  reliably  evaluate  it  is  determined  to  a  great  extent  by  the  established  information 
flows  within  an  organization  and  among  related  organizations.  Information  flows  are  concerned 
with  the  storage,  access,  and  transmission  of  information.  This  may  include  a  variety  of 
information  systems  to  which  analysts  have  access,  such  as  networks  and  databases,  chains  of 
communication  and  available  communication  means,  and  may  include  formally  established 
systems,  flows,  and  reporting  chains  as  well  as  informal  networks.  Intelligence  assessments  are 
often  time-sensitive,  and  therefore  it  is  important  for  analysts  to  have  access  to  relevant 
information  in  a  timely  manner.  Analyzing  information  requirements  and  current  flows  within  an 
intelligence  unit  will  help  to  identify  any  existing  discrepancies  between  the  requirements  and 
capabilities,  which  may  positively  impact  a  unit’s  functioning  if  resolved. 

4.2.2  Calibration  study  of  source  reliability  and  information  credibility 
assessments  by  information  collectors 

In  addition  to  gaining  timely  access  to  relevant  information,  analysts  need  to  have  reliable  means 
to  evaluate  the  quality  of  the  available  information.  As  we  discussed  above,  intelligence  analysts 
do  not  collect  “raw”  information  first-hand  and  have  to  rely  on  information  collected  and 
processed  by  others  (Pritchard  and  Goodman  2009).  Therefore,  analysts  may  need  to  rely  on  other 
people’s  assessments  of  information  reliability  and  accuracy.  Each  piece  of  information  collected 
through  specialized  intelligence  means  is  rated  on  two  dimensions:  reliability  of  the  source  and 
credibility  of  information.  Some  managers  commented  that  these  evaluations  are  not  always  done 
in  a  timely  manner  or  updated  when  changes  are  required,  especially  for  HUMINT  sources. 

In  order  to  gauge  the  accuracy  of  the  assessments  for  source  reliability  and  information  credibility 
a  calibration  study  of  these  assessments  may  be  undertaken.  Research  conducted  by  Mandel 
(2009b)  on  calibration  of  the  accuracy  of  analysts’  judgments  as  discussed  in  Sub-section  3.1.9 
could  be  extended  to  gauge  the  calibration  of  information  collectors  and  handlers  of  HUMINT 
sources  in  assessing  accuracy  of  source  reliability  and  information  credibility.  Similar  to 
determining  the  accuracy  of  analytic  judgments,  the  accuracy  of  certain  pieces  of  information 
provided  by  HUMINT  sources  may  be  evaluated  at  a  later  point  in  time.  A  dataset  can  be 
designed  that  consists  of  a  number  of  pieces  of  information  gathered  by  a  source  handler, 
reliability  and  credibility  evaluations  for  each  piece  of  information  assigned  by  the  source 
handler,  and  a  rating  of  whether  each  piece  of  information  was  accurate  or  not.  Similar  to  a 
calibration  analysis  of  analysts’  judgment  conducted  by  Mandel  (2009b),  a  calibration  analysis  of 
source  handlers’  assessments  of  sources  and  information  could  be  performed.  Such  a  calibration 
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study  would  provide  feedback  to  information  collectors  on  their  overall  accuracy,  and  would 
provide  more  information  and  justification  to  analysts  who  rely  on  these  assessments. 

4.2.3  Information  management:  Search,  aggregation,  and  collation  of 
information 

Technological  progress  has  significantly  affected  the  information  world  around  us.  The  amount  of 
available  information  has  grown;  the  variety  of  sources  from  which  information  could  be  obtained 
has  increased,  and  the  speed  with  which  information  can  be  transmitted  has  also  increased 
drastically.  More  and  more  information  is  available  to  analysts  from  various  open  sources  in 
addition  to  information  specifically  collected  through  various  intelligence  means. 

Intelligence  analysts  are  faced  with  a  constantly  increasing  amount  of  information,  but  the 
quantity  of  available  information  does  not  translate  necessarily  into  quality  and  reliability.  It  is 
virtually  impossible  for  analysts  to  undertake  exhaustive  searches  for  information  on  an  issue  as 
there  are  usually  time  constraints  to  which  they  need  to  be  mindful.  Without  conducting  an 
exhaustive  search,  analysts  cannot  know  what  other  information  may  be  available;  in  other  words, 
they  have  no  means  of  determining  whether  or  not  they  have  overlooked  an  important  piece  of 
information  in  their  search.  The  effectiveness  of  analysts’  searches  depends  to  a  great  extent  on 
the  design  and  management  of  various  information  systems  that  analysts  use.  The  effectiveness  of 
information  management  systems  in  turn,  requires  a  timely  and  thorough  collation  of  new 
information  and  the  development  of  a  comprehensive  ontology  to  allow  expedited  searches  of 
relevant  information. 

In  addition  to  the  examination  and  potential  improvement  of  existing  information  systems  and 
collation  practices,  it  may  be  beneficial  to  develop  training  programs  for  analysts  on  effective 
information  search  strategies  and  query  formulation  as  Patterson  et  al.  (2001)  suggested. 

4.3  Study  of  consumer-producer  relationships 

Intelligence  consumers  have  an  important  role  to  play  in  the  process  of  intelligence  production. 
Intelligence  producers  depend  on  their  consumers,  to  some  extent,  for  direction,  requirements, 
and  feedback  on  the  impact  of  their  intelligence  products.  In  order  to  be  relevant,  intelligence 
producers  -  analysts  and  their  managers  -  need  to  be  able  to  accurately  gauge  consumers’  current 
requirements  and  anticipate  their  potential  future  requirements.  Feedback  from  consumers  is 
essential  for  intelligence  organizations  in  directing  their  efforts  and  making  their  products 
relevant  and  usable.  The  two  groups  operate  in  different  organizational  environments  and  are 
subjected  to  different  pressures  and  constraints;  and  although  there  is  a  great  deal  of  dependency 
of  producers  on  consumers,  there  do  not  seem  to  be  many  opportunities  for  interaction.  The  two 
groups  may  differ  in: 

•  their  perceptions  of  the  role  that  intelligence  products  ought  to  play  in  consumers’  decision 
making, 

•  their  perceptions  of  the  characteristics  and  properties  of  a  good  intelligence  product, 

•  the  constraints  they  must  face,  and 

•  the  requirements  of  their  environments. 
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A  closer  examination  of  the  current  dynamics  of  consumer-producer  relationships  in  the  Canadian 
context  may  be  instrumental  in  improving  the  understanding  of  the  existing  issues,  their  causes, 
and  potential  ways  to  bridge  gaps.  However,  the  success  of  such  an  investigation  would  greatly 
depend  on  the  interest  and  input  from  the  intelligence  community  and  their  consumers. 

4.4  Knowledge  preservation  and  transfer  practices 

Under  conditions  of  high  personnel  turnover,  analysts  frequently  leave  the  organization  taking 
with  them  valuable  knowledge  and  experience.  Training  new  analysts  is  costly  and  time 
consuming  as  there  seems  to  be  no  established  means  to  transfer  knowledge  accumulated  by 
departing  analysts  to  the  new  arrivals.  This  issue  is  not  unique  to  the  Canadian  intelligence 
community  as  Treverton  and  Gabbard  (2008)  observed  similar  issues  in  the  US  community. 
Furthermore,  it  is  not  unique  to  the  domain  of  intelligence  analysis  as  many  companies  in 
industry,  in  addition  to  the  usual  turnover  rate,  have  to  deal  with  an  aging  workforce  and  the 
retirement  of  a  substantial  proportion  of  their  experts.  Implementing  effective  knowledge 
preservation  and  transfer  practices  has  the  potential  to  reduce  disruptions  caused  by  the  constant 
turnover  of  military  personnel,  reduce  the  learning  curve  for  new  analysts,  and  ensure  continuity. 
A  review  of  the  existing  best  practices  in  knowledge  preservation  and  transfer  in  industry,  along 
with  ways  and  means  to  implement  them,  may  be  valuable  to  the  intelligence  community. 

4.5  Selection  and  assessment  criteria 

Research  from  the  social  sciences  could  be  utilized  to  inform  selection  processes  for  intelligence 
analysts.  Research  in  this  domain  could  take  two  directions:  identifying  stable  individual 
difference  characteristics  that  predict  performance,  and  developing  tools  to  assess  them. 
Naturally,  research  in  this  area  will  be  closely  related  to  the  development  of  valid  measures  of 
analysts’  performance,  which  we  discuss  in  Section  4.7  below. 

As  some  managers  pointed  out,  in  addition  to  certain  skills  that  may  be  acquired  through  training 
and  experience,  successful  intelligence  analysts  possess  a  certain  set  of  inherent  characteristics.  It 
would  be  instrumental  to  the  optimal  selection  of  intelligence  analysts  to  be  able  to  identify  these 
inherent  traits.  One  way  to  do  so  would  be  to  assess  stable  individual  differences  (e.g.,  the  Five 
Factor  Model,  and  the  NFC  and  the  NFCC  scales)  among  analysts,  and  then  compare  them  to 
both  subjective  performance  evaluations  and  objective  judgment  accuracy  measures,  as  described 
in  Mandel  (2009b).  Identifying  traits  that  reliably  predict  sound  analysis  would  benefit  selection 
procedures  for  analysts. 

In  addition  to  identifying  and  assessing  stable  traits  on  which  to  select  analysts,  it  would  be  useful 
to  examine  skill  sets  that  predict  performance.  New  assessment  measures  that  are  uniquely 
tailored  to  the  analyst  profession  could  be  developed  in  close  collaboration  with  intelligence 
managers.  In  developing  these  measures,  the  set  of  essential  skills  and  capabilities  that  managers 
identified  could  be  taken  as  a  starting  point,  which  could  then  be  refined  upon  further  discussion 
with  intelligence  professionals.  As  part  of  the  selection  process,  candidate  intelligence  analysts 
could  be  assessed  on  the  identified  dimensions.  There  have  been  some  efforts  in  the  intelligence 
community  devoted  to  identifying  intelligence  analysts’  skill  sets  (e.g.,  Johnston  2005,  Moore  and 
Krizan  2003,  Moore  2005,  Moore  et  al.  2005,  Moore  2007);  however,  these  have  not  been 
correlated  with  analysts’  performance. 
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4.6  Training 


As  we  noted  above,  one  of  the  challenges  in  intelligence  analysis  training  is  evaluating  its 
effectiveness.  An  organization  such  as  DRDC  could  provide  support  to  the  intelligence 
community  in  designing  and  administering  comprehensive  and  objective  evaluations  of  current 
training  programs.  To  do  so,  DRDC  could  consult  with  course  instructors  to  identify  training 
objectives,  and  to  develop  tools  to  assess  the  degree  to  which  these  objectives  are  being  met. 
Initial  efforts  in  this  direction  are  currently  being  undertaken  by  members  of  TRIG  in 
collaboration  with  the  Canadian  Forces  School  for  Military  Intelligence  (CFSMI).  One  aspect  of 
the  research  program  evaluates  the  effectiveness  of  cultural  sensitivity  training  developed  by 
CFSMI  instructors  for  reducing  bias  in  attitudes  toward  an  out-group  (e.g.,  Arab-Muslims). 
Explicit  (i.e.,  self-report)  and  implicit  (i.e.,  sub-conscious  reaction-timed)  measures  of  attitudes 
are  administered  before  and  after  training  to  empirically  assess  the  impact  of  the  training  attitudes 
toward  the  out-group  (McLellan  et  al.  2010b).  Showing  a  preference  for  one  group  over  another 
has  implications  when  making  judgments  and  decisions  that  involve  members  of  these  groups. 

As  previously  mentioned,  intelligence  analysis  is  predominantly  a  cognitive  activity,  and 
incorporating  the  most  recent  findings  in  cognitive  and  behavioural  sciences  into  training  will 
provide  analysts  an  opportunity  to  utilise  this  knowledge  in  practice.  Keeping  intelligence  training 
up  to  date  with  recent  developments  in  behavioural  sciences  may  be  taxing  on  intelligence 
organizations’  resources,  and  collaboration  with  the  scientific  community  can  be  advantageous 
(Mandel  2009c).  Drawing  on  expertise  in  the  social  sciences,  DRDC  may  be  instrumental  in 
identifying  needs  and  developing  new  training  modules  for  intelligence  professionals.  For 
example,  one  of  the  authors  (Mandel)  utilised  his  expertise  in  behavioural  decision  research  to 
design  and  teach  a  one-day  course  on  judgment  and  decision  making  under  uncertainty  offered  to 
intelligence  analysts  in  the  context  of  IALP  training  program.  In  addition,  Mandel  developed  a 
training  module  to  improve  Bayesian  reasoning  with  visual  representations  of  probabilistic 
information  (Mandel  2009a).  The  training  has  been  extended  to  include  several  representation 
methods,  which  have  been  tested  on  the  general  population  with  positive  results  (Derbentseva  and 
Mandel  2010a).  In  addition,  the  training  has  been  shown  to  improve  the  posterior  probability 
judgment  accuracy  of  intelligence  analyst  trainees  at  CFSMI  (Mandel  2009a).  More  generally,  the 
behavioural  science  community  can  provide  valuable  support  to  the  IC  in  developing  and 
evaluating  intelligence  analysis  training. 

4.7  Performance  evaluation 

Performance  of  intelligence  analysts  is  greatly  contingent  on  the  quality  of  their  intelligence 
assessments.  As  we  discussed  in  Sub-section  3.1.9,  the  quality  of  intelligence  assessments  may  be 
evaluated  by  examining  two  main  criteria:  the  intelligence  process  and  the  intelligence  product 
(Moore  and  Krizan  2003).  Currently,  performance  on  both  criteria  is  evaluated  mostly  according 
to  managers’  subjective  judgment. 
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4.7.1  Evaluating  intelligence  products 


Although  a  set  of  criteria  as  outlined  by  Brei  (2005)  and  Moore  and  Krizan  (2003)  -  usability, 
relevance,  and  timeliness  -  may  be  used  in  assessing  intelligence  products,  these  criteria  may  not 
fully  capture  the  value  added  by  an  intelligence  product.  A  product’s  quality  needs  to  be  assessed, 
but  “quality”  is  a  complex  and  multi-faceted  term. 

Investigating  the  meaning  of  the  "quality"  of  a  report  to  different  stakeholders  (i.e.,  consumers, 
managers,  and  analysts)  may  be  informative  in  revealing  differences  in  the  perceived  value  of 
intelligence  assessments.  Such  an  exercise  could  also  facilitate  the  articulation  of  other  criteria  for 
assessing  the  quality  of  intelligence  products,  in  addition  to  those  identified  by  Brei  (2005). 

Calibration  of  analysts’  judgments  accuracy  may  be  another  criterion  for  evaluating  intelligence 
reports.  Working  in  an  environment  characterized  by  uncertainty,  analysts  are  bound  to  make 
incorrect  judgments  at  least  some  of  the  time.  For  this  reason,  the  ability  of  analysts  to  accurately 
estimate  confidence  in  their  judgments  (i.e.,  to  be  well  calibrated)  is  paramount.  To  be  well 
calibrated  is  to  have  insight  into  one’s  own  thought  processes  and  to  be  well-attuned  to  factors 
(both  internal  and  external)  that  affect  the  likelihood  that  one’s  predicted  estimate  will 
occur.  When  analysts  can  accurately  communicate  confidence  in  their  judgments,  stakeholders 
can  use  this  information  when  making  decisions,  especially  those  that  are  contingent  on  the 
predicted  outcome. 

4.7.2  The  intelligence  process:  Investigating  the  feasibility  of 
developing  a  standard  “Challenging  mechanism” 

Some  managers  commented  that  they  find  it  difficult  to  challenge  the  thinking  process  and  the 
quality  of  analysis  of  their  analysts  without  having  sufficient  background  on  the  issue  being 
analysed.  In  order  to  ease  this  process,  some  managers  suggested  that  it  may  be  possible  to 
develop  a  standard  challenging  mechanism  that  could  be  applied  to  evaluate  the  soundness  of 
analysis  regardless  of  the  evaluator’s  background  knowledge  on  the  issue.  Such  a  mechanism, 
would  address  information  selection  and  interpretation,  assess  logic  and  thinking,  examine  the 
validity  of  underlying  assumptions,  and  probe  for  potential  distortions  due  to  biases.  It  is 
worthwhile  to  investigate  the  possibility  of  developing  such  a  generic  evaluation  mechanism.  A 
deep  understanding  of  the  intelligence  analysis  process  and  potential  issues  will  be  essential  in 
this  effort.  However,  if  the  effort  is  successful,  the  resulting  mechanism  may  be  used  by  both 
managers  and  analysts  themselves  to  improve  the  soundness  of  their  analyses. 
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5  Concluding  remarks 


In  the  context  of  the  year  for  scoping  the  “Understanding  and  Augmenting  Human  Capabilities  in 
Intelligence  Production”  ARP,  we  conducted  interviews  with  intelligence  managers  from  two 
Canadian  intelligence  organizations:  CDI  and  IAS.  The  aim  of  the  interviews  was  to  identify  the 
existing  issues  and  areas  for  further  R&D  in  the  social  and  cognitive  science  domains  that  may  be 
beneficial  to  the  intelligence  community.  In  this  report,  we  summarized  and  discussed  the  issues 
raised  by  the  managers  during  the  interviews  and  offered  potential  areas  of  further  research. 

We  have  identified  a  number  of  areas  where  further  scientific  research  may  be  of  value  to  the 
Canadian  intelligence  community.  These  include:  current  analytic  methods  employed  in  the 
community  and  evaluation  of  their  effectiveness;  cognitive  processes  involved  in  intelligence 
analysis;  validation  of  various  analytic  tools;  investigation  of  information  dependencies,  flows 
and  breakdowns;  development  and  evaluation  of  training;  investigation  of  stable  individual 
differences  and  capabilities  that  are  essential  for  intelligence  analysis;  knowledge  preservation, 
management,  and  transfer;  issues,  perceptions,  and  dynamics  in  consumer-producer  relationships; 
and  issues  in  performance  and  intelligence  product  evaluation. 

Bruce  and  George  (2008)  argued  that,  although  thousands  of  professionals  practice  the  “craft”  of 
intelligence  analysis  daily,  the  professional  discipline  of  intelligence  analysis  and  what  constitutes 
good  analytic  principles  and  practices  remains  largely  undefined.  Intelligence  analysis  has  only 
recently  come  into  the  centre  of  attention  and  (unclassified)  scientific  research  literature  devoted 
to  it  is  still  rather  scarce.  With  a  gradually  growing  body  of  dedicated  literature,  intelligence 
analysis  is  only  beginning  to  undergo  its  professional  formation  (Bruce  and  George  2008,  Fisher 
and  Johnston  2008).  There  are  important  opportunities  for  the  social  and  cognitive  sciences  to 
make  contributions  to  this  field  and  its  development.  The  identified  areas  for  R&D  encompass 
many  issues  and  span  over  several  disciplines.  Clearly,  it  would  be  impossible  to  address  all  of 
the  identified  questions  within  the  scope  of  a  single  project.  The  priorities  for  future  work  need  to 
be  identified  through  continuous  and  close  interaction  with  members  of  the  Canadian  IC. 
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Annex  A 


Initial  List  of  Interview  Questions 


A.1  Primary  list  of  questions 

A.1.1  Managers’  Background 

•  Could  you  tell  us  about  your  educational  and  career  background? 

•  How  long  have  you  worked  at  IAS  (CDI)? 

•  How  long  have  you  been  in  the  manager  role  at  IAS  (CDI)? 

•  How  many  analysts  currently  report  to  you? 

•  All  together,  how  many  have  you  managed  in  the  past? 

•  Were  you  an  intelligence  analyst  at  some  point  in  your  career?  If  yes,  for  how  long? 

A.1. 2  Analyst  Tasks 

•  Could  you  tell  us  about  the  range  of  tasks  that  analysts  reporting  to  you  typically  have  to 
perform? 

•  Could  you  describe  the  steps,  cycles,  or  processes  that  are  involved  in  completing  these 
tasks? 

•  What  are  the  major  difficulties,  if  any,  associated  with  each  task? 

•  What  are  the  forms  that  analysts’  products  tend  to  take? 

a.  For  reports:  are  these:  mostly  qualitative;  mostly  quantitative;  or  roughly  an  even 
balance  of  the  two? 

b.  Are  they  mostly  descriptive  /  explanatory  /  predictive? 

•  What  are  the  types  of  queries  posed  to  analysts? 

•  What  are  the  typical  timelines  for  assessment  completion? 

•  Do  analysts  work  on  a  single  report  at  a  time  or  do  they  work  on  multiple  reports 
simultaneously? 

•  How  do  analysts  deal  with  issues  regarding  the  communication  of  uncertainty  of  events  in 
their  reports? 

a.  Are  there  standard  procedures  in  place  for  assessing  and  communicating  uncertainty? 

If  not,  do  you  think  this  an  issue  that  needs  more  attention? 
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A. 1.3  Analyst  Capability 

•  What  is  the  selection  procedure  for  analysts?  Are  there  cognitive  skills  or  trait  qualities  that 
are  tested? 

•  What  level  of  formal  education  and  career  backgrounds  do  analysts  typically  have?  What 
are  the  predominant  disciplines  from  which  they  are  recruited? 

•  What  intelligence  analysis  training  have  the  analysts  reporting  to  you  received? 

•  Do  analysts  learn  about  issues  such  as  cognitive  biases  and  mindsets  that  can  impair  their 
judgment? 

a.  If  so,  about  which  biases  and  mindsets  do  they  learn? 

b.  Do  they  learn  about  ways  of  mitigating  the  impact  of  mindsets  and  biases? 

c.  Are  there  tools  or  procedures  in  place  that  are  specifically  designed  to  correct  for 
these  biases? 

•  Is  the  effectiveness  of  training  evaluated? 

a.  If  so,  how  regularly? 

b.  In  what  manner? 

c.  What  have  such  evaluations  revealed  about  the  adequacy  of  training? 

d.  How  do  you  think  training  might  be  feasibly  improved? 

•  How  is  analyst  performance  evaluated?  (i.e.,  what  are  the  criteria?) 

a.  Do  analysts  receive  feedback  and  training  support  based  on  those  evaluations? 

•  In  your  view,  what  are  the  skills  and  capabilities  that  analysts  bring  to  the  job  that  are  most 
important  in  being  effective  as  an  analyst? 

•  In  your  view,  what  are  the  greatest  challenges  to  human  capability  development  for 
intelligence  analysts? 

•  Are  there  specific  tools  that  analysts  use? 

a.  If  so,  which  ones? 

b.  What  are  their  advantages  and  disadvantages? 

c.  Are  analysts  required  to  use  them? 

•  A  number  of  issues  have  been  raised  regarding  the  ability  of  analysts  to  produce  good 
intelligence;  of  these,  which  would  you  identify  as  the  top  3  concerns  that  ought  to  be 
addressed? 
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A. 1.4  Management  Tasks 

•  Could  you  describe  the  main  functions  and  tasks  that  you  perform  as  a  manager? 

•  Could  you  describe  the  steps,  cycles,  or  processes  that  are  involved  in  completing  these 
tasks?  Given  that  these  may  differ  for  the  different  major  tasks  you  perform,  please  feel  free 
to  expand  on  each  task  separately. 

•  What  are  the  major  difficulties,  if  any,  associated  with  each  task? 

•  Are  some  tasks  more  challenging  than  others?  If  so,  what  makes  them  more  challenging? 

A.1.5  Management  Capability 

•  In  your  view,  what  are  the  skills  and  capabilities  that  you  bring  to  the  job  that  you  believe 
are  most  important  in  being  effective  as  a  manager?  Given  that  these  may  differ  for  the 
different  major  tasks  you  perform,  please  feel  free  to  expand  on  each  task  separately. 

•  Are  there  other  skills  and  capabilities  that  you  think  are  particularly  important  for  managers 
of  intelligence  organizations  to  have? 

•  In  your  view,  what  are  the  greatest  challenges  to  human  capability  development  for 
managers  of  intelligence  organizations? 

•  What  do  you  think  might  be  done  to  meet  these  challenges-that  is,  to  improve  upon  human 
capabilities  for  the  management  of  intelligence  analysis? 

•  Can  you  think  of  any  other  information  that  we  haven’t  covered  today  that  you  think  would 
be  useful  to  our  research?  (e.g.,  intelligence  issues,  resources,  points  of  contact) 

A.2  Additional  (secondary)  questions 

A.2.1  Manager’s  Background 

•  Could  you  tell  us  about  the  types  of  management  training  you  may  have  received? 

•  What  training  did  you  receive  as  an  analyst? 

A.2.2  Analyst  Task 

•  Are  the  analysts’  products  mostly  of  tactical  /  operational  /  strategic  importance? 

•  Are  analysts  usually  specialists  (e.g.,  in  SIGINT)  or  generalists? 

•  To  what  extent  do  analysts  draw  from  a  variety  of  different  sources  of  information  (e.g. 
HUMINT,  SIGINT,  IMINT,  COMINT,  OSINT  etc.)?  Which  sources  tend  to  be  most 
important  or  most  often  used  for  the  types  of  intelligence  assessments  made  in  your 
organization? 

A.2.3  Analyst  Capability 

•  What  is  your  view  on  professionalization  of  the  intelligence  community? 
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Annex  B  Taxonomy  of  Problem  Types7 


Characteristics 

Problem  Types 

Simplistic 

Deterministic 

Moderately 

Random 

Severely 

Random 

Indeterminate 

What  is  the 
question 

Obtain 

information 

How  much? 
How  many? 

Identify  and 
rank  all 

outcomes 

Identify  outcomes 
in  unbounded 

situation 

Predict  future 
events/situatio 

ns 

Role  of  facts 

Highest 

High 

Moderate 

Low 

Lowest 

Role  of 

judgment 

Lowest 

Low 

Moderate 

High 

Highest 

Analytical  tasks 

Find 

information 

Find/create 

formula 

Generate  all 

outcomes 

Define  potential 
outcomes 

Define  future 
factors 

Analytical 

method 

Search 

sources 

Match  data  to 
formula 

Decision 
theory;  utility 
analysis 

Role  playing  and 
gaming 

Analyse 
models  and 

scenarios 

Analytical 

instrument 

Matching 

Mathematical 

formula 

Influence 

diagram, 

utility, 

probability 

Subjective 
evaluation  of 

outcomes 

Use  of  experts 

Analytical 

output 

Fact 

Specific  value 
or  number 

Weighted 

alternative 

outcomes 

Plausible 

outcomes 

Elaboration  on 
expected  future 

Probability  of 
error 

Lowest 

Very  low 

Dependent  on 
data  quality 

High  to  very  high 

Highest 

Follow  up  task 

None 

None 

Monitor  for 

change 

Repeated  testing 
to  determine  true 

state 

Exhaustive 

learning 

7  From  Krizan,  L.  (1999).  Intelligence  Essentials  for  Everyone.  Joint  Military  Intelligence  College. 
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Annex  C  Concept  Map  outlining  issues  in  intelligence  analysis 
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List  of  symbols/abbreviations/acronyms/initialisms 


ACH 

Analysis  of  Competing  Hypotheses 

AIS 

Adversarial  Intent  Section 

ARP 

Applied  Research  Program 

CAPIA 

Canadian  Association  of  Professional  Intelligence  Analysts 

CDI 

Chief  of  Defence  Intelligence 

CFSMI 

Canadian  Forces  School  of  Military  Intelligence 

CIA 

Central  Intelligence  Agency 

CLT 

Construal  Level  Theory 

CMap 

Concept  Map 

COI POI 

Community  of  Interest  on  the  Practice  and  Organization  of  Intelligence 

COMINT 

Communications  Intelligence 

DIA 

Defence  Intelligence  Agency 

DGMPRA 

Director  General  Military  Personnel  Research  &  Analysis 

DRDC 

Defence  Research  &  Development  Canada 

GFF 

Global  Futures  Forum 

HREC 

Human  Research  Ethics  Committee 

HUMINT 

Human  Intelligence 

IACC 

Intelligence  Assessment  Coordination  Committee 

IALP 

Intelligence  Assessment  Learning  Program 

IAS 

International  Assessment  Staff 

IC 

Intelligence  Community 

IEG 

Interdepartmental  Expert  Group 

IMINT 

Image  Intelligence 

ITAC 

Integrated  Threat  Assessment  Centre 

MACE 

Method  for  Assessing  the  Credibility  of  Evidence 

NFC 

Need  for  Cognition 

NFCC 

Need  for  Cognitive  Closure 

NSA 

National  Security  Agency 

ODNI 

The  Office  of  the  Director  of  National  Intelligence 

OSINT 

Open  Source  Intelligence 

PCO 

Privy  Council  Office 

R&D 

Research  &  Development 

ROI 

Return  On  Investment 

SIGINT 

Signals  Intelligence 

TRIG 

Thinking,  Risk,  and  Intelligence  Group 

TSD-PI 

Trait  Self-Descriptive  Personality  Inventory  Revisited  measure 

UK 

United  Kingdom 

US 

United  States 

WMD 

Weapons  of  Mass  Destruction 
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